[netflow-tools] softflowd -m 512000 ... flow-capture ... 90 % less traffic??
Sebastian Majkowski
smajko at wp.pl
Fri Aug 14 18:09:35 EST 2009
Andrew McGill wrote:
> Greetings netflow-tools,
>
> I have softflowd sending information to flow-capture for a network with a few
> hundred hosts (don't ask, the answer is probably "yes"). Softflowd was
> configured with the default without a -m parameter, so that softflowd tracked
> a maximum of 8192 flows. The primary reason for rolling over flows was
> running out of connections - and cpu load was obnoxiously high. So I fixed it
> (in the sense of thereifixedit.com, perhaps). I told softflowd that it should
> track a maximum of 512000 flows, and it duly did.
>
> The before and after log files for 10 minutes of traffic look like this:
>
> -rw-r--r-- 1 root root 12678211 Jul 26 17:02 ft-v05.2009-07-26.165257+0200
> -rw-r--r-- 1 root root 673952 Jul 26 17:32 ft-v05.2009-07-26.172247+0200
>
> ... which is great, BUT it seems that most of the traffic is getting lost.
> It's not that this traffic is getting deferred into later stats -- it simply
> never gets reported -- the reported totals dropped to 10% of their previous
> values!
>
> before: Average Kbits / second (real) : 49598.9333
> after: Average Kbits / second (real) : 3872.6817
>
> The next day it was still roughly 10% of the real amount:
>
> Average Kbits / second (real) : 4617.1089
>
> Is this correct behaviour? Am I doing one or more things wrong?
>
> &:-)
>
>
>
> Notes:
>
> Startup parameters:
> flow-capture -p /var/run/flow-capture.pid -n 144 -N -1 \
> -w /var/log/netflows -S 10 0/0/8828
>
> softflowd -i eth2 -n 127.0.0.1:8828 # BEFORE
> softflowd -i eth2 -n 127.0.0.1:8828 -m 512000 # AFTER
>
>
> In case it's relevant, this is what flow-stat said about the files:
>
>
> # --- ---- ---- Report Information --- --- --- (BEFORE)
> #
> # Fields: Total
> # Symbols: Disabled
> # Sorting: None
> # Name: Overall Summary
> #
> # Args: flow-stat
> #
> Total Flows : 723704
> Total Octets : 2975935893
> Total Packets : 6138299
> Total Time (1/1000 secs) (flows): 5790296389
> Duration of data (realtime) : 480
> Duration of data (1/1000 secs) : 2363291
> Average flow time (1/1000 secs) : 8000.9183
> Average packet size (octets) : 484.8144
> Average flow size (octets) : 4112.0900
> Average packets per flow : 8.4818
> Average flows / second (flow) : 306.2649
> Average flows / second (real) : 1507.7167
> Average Kbits / second (flow) : 10075.1113
> Average Kbits / second (real) : 49598.9333
>
>
> IP packet size distribution:
> 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
> .000 .379 .286 .090 .082 .044 .024 .021 .013 .007 .005 .003 .003 .001 .002
>
> 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
> .003 .005 .001 .014 .018 .000 .000 .000 .000 .000 .000
>
> Packets per flow distribution:
> 1 2 4 8 12 16 20 24 28 32 36 40 44 48 52
> .643 .086 .075 .097 .030 .018 .011 .007 .005 .004 .003 .002 .002 .002 .001
>
> 60 100 200 300 400 500 600 700 800 900 >900
> .002 .005 .003 .001 .001 .000 .000 .000 .000 .000 .001
>
> Octets per flow distribution:
> 32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 8192
> .000 .241 .298 .191 .104 .082 .022 .011 .007 .004 .004 .002 .002 .002 .002
>
> 8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872
> .001 .001 .001 .001 .001 .001 .001 .001 .001 .001 .019
>
> Flow time distribution:
> 10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
> .733 .014 .013 .024 .035 .032 .025 .017 .010 .008 .016 .007 .005 .005 .004
>
> 12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000
> .006 .004 .003 .003 .002 .004 .002 .001 .001 .001 .023
>
> # --- ---- ---- Report Information --- --- --- (AFTER)
> #
> # Fields: Total
> # Symbols: Disabled
> # Sorting: None
> # Name: Overall Summary
> #
> # Args: flow-stat
> #
> Total Flows : 50516
> Total Octets : 261406012
> Total Packets : 551158
> Total Time (1/1000 secs) (flows): 329152148
> Duration of data (realtime) : 540
> Duration of data (1/1000 secs) : 1366814
> Average flow time (1/1000 secs) : 6515.8001
> Average packet size (octets) : 474.2851
> Average flow size (octets) : 5174.7172
> Average packets per flow : 10.9106
> Average flows / second (flow) : 36.9810
> Average flows / second (real) : 93.5481
> Average Kbits / second (flow) : 1530.9283
> Average Kbits / second (real) : 3872.6817
>
>
> IP packet size distribution:
> 1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
> .000 .205 .364 .125 .116 .053 .028 .020 .019 .009 .005 .004 .003 .002 .002
>
> 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
> .003 .008 .001 .015 .019 .000 .000 .000 .000 .000 .000
>
> Packets per flow distribution:
> 1 2 4 8 12 16 20 24 28 32 36 40 44 48 52
> .439 .174 .073 .119 .067 .034 .018 .015 .010 .008 .006 .004 .003 .004 .002
>
> 60 100 200 300 400 500 600 700 800 900 >900
> .004 .010 .005 .001 .001 .000 .000 .000 .000 .000 .001
>
> Octets per flow distribution:
> 32 64 128 256 512 1280 2048 2816 3584 4352 5120 5888 6656 7424 8192
> .000 .059 .297 .200 .153 .154 .042 .022 .011 .006 .004 .003 .002 .004 .003
>
> 8960 9728 10496 11264 12032 12800 13568 14336 15104 15872 >15872
> .002 .002 .002 .001 .001 .001 .001 .001 .001 .000 .027
>
> Flow time distribution:
> 10 50 100 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
> .298 .024 .012 .073 .319 .071 .035 .024 .012 .010 .020 .009 .007 .006 .005
>
> 12000 14000 16000 18000 20000 22000 24000 26000 28000 30000 >30000
> .009 .005 .003 .004 .004 .003 .003 .003 .003 .004 .035
>
>
>
>
> _______________________________________________
> netflow-tools mailing list
> netflow-tools at mindrot.org
> https://lists.mindrot.org/mailman/listinfo/netflow-tools
>
>
>
>
Hi Andrew,
I am not an expert in net-flow but I had similar situation. I guess that
this is the result of increasing max flows tracked - so less frequently
netflow records will be created (and smaller file). When maximum 8192
flows was reached probably softflowd just ends some flows creating
records to manage other flows, thats why the file is bigger. The same
situation is when manipulating timers - this allows you to decide when
(or how long) the flow is tracked before creating netflow record...
Maybe it would be good to trace some connections from one of you users
to see how they are placed in netflow records - this will prove if the
data is tracked or not (as you suspect)
Regards,
Sebastian
More information about the netflow-tools
mailing list