[netflow-tools] softflowd under FreeBSD measures too much traffic

[Sebastian Schwerdhoefer]

Damien Miller wrote:
> Not without a debug trace from softflowd and a better description of
> how you calculated a variance.

Excuse my stupid question, but I must admit that I'm not familar with
debugging. Also I'm not a C programmer, so I need additional help to create a
debug trace. How can I create a debug trace?

The other thing however, the better explanation should work :-)
I'll do a bit of ascii art to show my test setup:

            sis0  sis1
+--------+    +----+    +--------+
| Host 1 |----| GW |----| Host 2 |
+--------+    +----+    +--------+
                 |    +------------+
                 `----| Accounting |
                      +------------+

Host 1, Host 2 and the accounting machine are in seperate networks. GW is a pc
that runs as ip gateway, to connect these networks. GW also runs softflowd which
sends v5 flows to "accounting". The accounting machine uses pmacct to receive
the netflow datagrams, calculate the total traffic between each host and save
the values in a postgres database. 

Sis0 is gw's nic to host 1's network, 
Sis1 is gw's nic to host 2's network.
Sis2 (not shown) is the nic to accounting's network.

I did several test, where I started softflowd, ipfm (as third-party accounting
tool, to compare the results) and tcpdump at (almost) the same time on the
gateway. Therefore I have the following little bash script:

startup.bash
    ipfm -c /tmp/ipfm.conf &
    softflowd -D -i sis1 -v 5 -n accounting:2100 &
    tcpdump -w /tmp/sis1.dump -i sis1 &

On Host 1 I type "scp host2:/boot/vmlinuz* .", to create some traffic that the
three apps shall measure.

Of course I have to stop all three processes at once. I wait a few minutes
(until softflowd detected expired flows and send the netflow datagrams to
accounting) and stop the measurement with:

shutdown.bash
    killall ipfm &
    killall softflowd &
    killall tcpdump &

If I compare the traffic that ipfm calculated with the values that pmacct wrote
to database, pmaccts values are always a little bit (not a data bit :-)) higher
than ipfm's results. I don't suspect pmacct as failure, because softflowd's
debug output shows the same values that pmacct writes to database.

Now the mystic part: I clear the postgres database and advise softflowd to read
the tcpdumped file with: softflowd -D -r /tmp/sis1.dump -v 5 -n
accounting:2100. Acracadabra, the values in the postgres database are the same
as the values from ipfm!

I did this test several times. The exact values do always differ, but the final
result (that softflowd measures more traffic when it's directly sniffing, but
exact values when it analyzes the tcpdump file) is always the same.
 
Thank you for reading and for your help!
... and excuse me for my skrewed up english :-/ I hope you can understand me...

best regards,
Sebastian Schwerdhöfer