[netflow-tools] flowd benchmark

[Damien Miller]

Gijs Molenaar wrote:
> Hello people,
> 
> I'm doing some research for what is the best flow analyse tool for us at 
> the moment. We have routers generating around the 1.000.000 flows every 
> 5 minutes, and this is already sampled with a rate of 100. So speed is 
> very important for us. The 2 tools I like the most are flowd and 
> flow-tools. Flowd supports v9 (and with that ipv6), so I prefer flowd.

(I assume that you are using flowd-0.8.5)

> The first thing that I was looking at was the load of the capture 
> daemon. There isn't a big difference between the 2. I use a quite slow 
> computer (pentium III 450, 1 GB ram), and both deaemons use about 10% 
> CPU time. When the PC is very busy, flow-tools (flow-capture) starts to 
> drop packages and logs this. My question is, what will happen with flowd 
> when the CPU load is too high to process a high flow of flows? The fact 
> that flows are dropped isn't important for us, but how many can be 
> interesting.

flowd doesn't detect if packets are dropped by the kernel before they
reach the daemon. It should check the netflow v5+ sequence numbers, and
this is already on the todo list.

> The next thing I did was flow analysation. I tried both python libraries 
> for this job. I captured 5 minutes with each daemon. Flowd will write 
> all info it has to the file, flow-tools does this also. The results 
> where stunning. These are the results (scripts are attached):
> 
> $ python flowtools.py
> finished in 20 seconds
> flowcount: 931711
> 45769 flows/s
> 
> $ python flowd.py
> finished in 256 seconds
> flowcount: 944281
> 3688 flows/s

Does turning off storing the CRC32 in flowd.conf speed this up?

flowd is always going to have to do a little more work, because the set
of fields that it stores is variable. That being said, it should be
possible to speed up the reader function by moving more it from the pure
python part of the module to the C implementation.

If I get time, I'll look at it on the weekend.

-d