[netflow-tools] flowd benchmark

[Gijs Molenaar]

Hello people,

I'm doing some research for what is the best flow analyse tool for us at 
the moment. We have routers generating around the 1.000.000 flows every 
5 minutes, and this is already sampled with a rate of 100. So speed is 
very important for us. The 2 tools I like the most are flowd and 
flow-tools. Flowd supports v9 (and with that ipv6), so I prefer flowd.

The first thing that I was looking at was the load of the capture 
daemon. There isn't a big difference between the 2. I use a quite slow 
computer (pentium III 450, 1 GB ram), and both deaemons use about 10% 
CPU time. When the PC is very busy, flow-tools (flow-capture) starts to 
drop packages and logs this. My question is, what will happen with flowd 
when the CPU load is too high to process a high flow of flows? The fact 
that flows are dropped isn't important for us, but how many can be 
interesting.

The next thing I did was flow analysation. I tried both python libraries 
for this job. I captured 5 minutes with each daemon. Flowd will write 
all info it has to the file, flow-tools does this also. The results 
where stunning. These are the results (scripts are attached):

$ python flowtools.py
finished in 20 seconds
flowcount: 931711
45769 flows/s

$ python flowd.py
finished in 256 seconds
flowcount: 944281
3688 flows/s

The flowd python library is about 12x slower! I was really not happy 
when I saw this output.

The thing is, I can't use flowd now. I need to do a _lot_ more 
computations than to calculate in and out AS traffic. Running flowtools 
python program on a (at the moment) fast machine can speed it up by 
about a factor 5, but then flowd would still be much to slow.

Maybe it has to do something with the fact that with flow-tools I do a 
readlines() to load the whole file in memory. With flowd it 'walks' 
trough the file, which can be much slower. But I'm not sure. flowtools 
python libary is also completely written in C.

I like to use flowd, so I wanted to try to change the flowtools python 
source to be able to read the flowd binary format. I'm not really a good 
C programmer, but I can give it a try :).


Greetings,

--
Gijs Molenaar
gijs at looze.net
http://gijs.looze.net

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: flowtools.py
Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050713/164659e0/attachment.ksh 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: flowd.py
Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050713/164659e0/attachment-0001.ksh