[netflow-tools] flowd benchmark

[Gijs Molenaar]

Damien Miller wrote:

> Damien Miller wrote:
>
>> flowd is always going to have to do a little more work, because the set
>> of fields that it stores is variable. That being said, it should be
>> possible to speed up the reader function by moving more it from the pure
>> python part of the module to the C implementation.
>
>
> OK, I moved all of the flow reding into the C part of the Python module
> and it didn't help much.

That's fast!  : )

I did a little more research. Because I thought the python API of flowd 
was slow, I wanted to write a flowd-reader parser in python. I tried 3 
flowcapture programs and their readers.

The test is the most basic operation, just read out the flow log file 
and print the fields


$ time flowd-reader ./flowd-log | wc -l
944282
real    1m55.109s
user    1m23.378s
sys     0m42.296s


$ time flow-export -f2 < ./flowtools-log | wc -l
flow-export: Exported 931711 records
931712
real    0m24.225s
user    0m23.384s
sys     0m2.037s

Much, faster but not variable field length.


$ time ./ipflow grep ./netflow.log | wc -l
1280538
real    1m45.336s
user    1m44.407s
sys     0m3.188s


This is a new one I tried, supporting v9. It isn't that much faster than 
flowd. So it really is the variable field length thing that makes it slow.

All tests where done with v5 cisco flows, and on a 2 proccessor system.