From djm at mindrot.org Thu Sep 3 18:22:07 2009 From: djm at mindrot.org (Damien Miller) Date: Thu, 3 Sep 2009 18:22:07 +1000 (EST) Subject: [netflow-tools] softflowd -m 512000 ... flow-capture ... 90 % less traffic?? In-Reply-To: <200908031748.07004.list2009@lunch.za.net> References: <200908031748.07004.list2009@lunch.za.net> Message-ID: On Mon, 3 Aug 2009, Andrew McGill wrote: > ... which is great, BUT it seems that most of the traffic is getting lost. > It's not that this traffic is getting deferred into later stats -- it simply > never gets reported -- the reported totals dropped to 10% of their previous > values! > > before: Average Kbits / second (real) : 49598.9333 > after: Average Kbits / second (real) : 3872.6817 > > The next day it was still roughly 10% of the real amount: > > Average Kbits / second (real) : 4617.1089 > > Is this correct behaviour? Am I doing one or more things wrong? Long-running flows may live in softflowd's flow tree indefinitely unless you specify a maxiumum lifetime (-t maxlife=N). If they don't expire then they will not be exported and therefore won't appear in your summarisation. 5 minutes is probably a good starting value. -d From djm at mindrot.org Thu Sep 3 18:26:01 2009 From: djm at mindrot.org (Damien Miller) Date: Thu, 3 Sep 2009 18:26:01 +1000 (EST) Subject: [netflow-tools] Reg softflowd v9 In-Reply-To: References: Message-ID: On Wed, 19 Aug 2009, Koteswar wrote: > Hi > I am runing softflowd in my PC with following command > #softflowd -D -v 9 -i eth1 -n192.168.100.2:5555 -t maxlife=10 -t expint=10 > And it is not sending v9 template flow set immadiately. If I create some > ICMP session with > #ping 192.168.100.254 -c 4 > then it is sending template flow set but not data flow set with the ICMP > session data. > If I do ping again then it is sending data flow set with the ICMP session > data. So we are missing first data flow set. I am attaching the debug info > with this mail. >From the start of your debuglog: > Finished scan 2 flow(s) to be evicted > Flow 1/0: r 0 offset 159 type 0004 len 35(0x0023) flows 1 > Flow 1/1: r 0 offset 190 type 0004 len 66(0x0042) flows 2 > Sending flow packet len = 192 This packet length certainly looks like a template flowset + two flows. Can you try to catch the problem with tcpdump? -d From list2009 at lunch.za.net Tue Sep 8 06:28:16 2009 From: list2009 at lunch.za.net (Andrew McGill) Date: Mon, 7 Sep 2009 22:28:16 +0200 Subject: [netflow-tools] Dealing with duplicates in softflowd Message-ID: <200909072228.16533.list2009@lunch.za.net> I've written some code for softflowd to deal (somewhat) with duplicate packets, in order to act more appropriately for bandwidth accounting. (See attached patch, if the list server doesn't eat it). Softflowd receives duplicates when: 0) Something transmits duplicates on the network (duh). 1) IPVS or something similar forwards a packet to another server for handling. The retransmitted packet is identical to the original packet, except for a modified set of MAC addresses. 2.0) Softflowd is attached to a mirror/span port, and sees a forwarded packet multiple times -- e.g. it is received externally, and received and transmitted internally. 2.1) A packet crosses multiple vlans in the course of its travels, all of which send copies to the mirror/span port. 2.2) The switch does some unicast flooding for reasons of its own, in which case, it may or may not grace the span/mirror port with multiple copies of its flood. (This stuff must be coming from somewhere - we see so much of it!) The attached patch adds the calculation of a CRC on the top and tail of the IP or IP6 part of a packet, and discards packets with matching CRC's as duplicates. For my purposes, I don't want to consider the MAC addresses as part of the calculation. It's not quite right yet ... * Is there a better crc32 function I can use? The random one I've hacked the code with is GPL licenced, which is doesn't quite fit with the current licence of softflowd. * An alternative to top and tail-ing the packet through crc32 for a quick and cheap hash would be worth considering. * I've added 66 bytes per flow (give or take who-knows-what for alignment) to struct FLOW for tracking the last few checksums. It would be nicer to allocate space for checksums dynamically, since it should be optional - although I dislike variable-sized arrays almost as much as extra pointers ... what difference is 33% extra memory between friends when an important topic like the elimination of duplicates is considered? Is there a neat way to do it in a single dynamically allocated array? -------------- next part -------------- A non-text attachment was scrubbed... Name: softflow-duplicates.patch Type: text/x-patch Size: 11199 bytes Desc: not available URL: From koti.kelam at gmail.com Tue Sep 8 20:25:25 2009 From: koti.kelam at gmail.com (Koteswar) Date: Tue, 8 Sep 2009 15:55:25 +0530 Subject: [netflow-tools] Listening on multiple interfaces in softflowd. Message-ID: Hi I want softflowd to listen on more than one interface. So should I start one instance of the daemon for one interface or is there any other way of spacifying all interfaces at once. Thanks in advance Koteswar -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean at tinfoilhat.ca Wed Sep 9 02:57:06 2009 From: sean at tinfoilhat.ca (Sean Cody) Date: Tue, 8 Sep 2009 09:57:06 -0700 Subject: [netflow-tools] Listening on multiple interfaces in softflowd. In-Reply-To: References: Message-ID: I have multiple instances exporting to separate sensors (helps keeps datasets isolated and easier for me to differentiate traffic paths). You could bridge all the interfaces together (assuming just listening) but that's kind of stupid so I would only personally suggest multiple instances. Then again I'm just picking this up as I go. -- Sean (mobile) On 2009-09-08, at 3:25 AM, Koteswar wrote: > Hi > > I want softflowd to listen on more than one interface. So should I > start one instance of the daemon for one interface or is there any > other way of spacifying all interfaces at once. > > Thanks in advance > Koteswar > _______________________________________________ > netflow-tools mailing list > netflow-tools at mindrot.org > https://lists.mindrot.org/mailman/listinfo/netflow-tools From kevin.wilcox at gmail.com Thu Sep 10 05:23:30 2009 From: kevin.wilcox at gmail.com (Kevin Wilcox) Date: Wed, 9 Sep 2009 15:23:30 -0400 Subject: [netflow-tools] failed exports Message-ID: <5d6848b00909091223g3aec6137q4f3f768845649aab@mail.gmail.com> Hi everyone. I'm running softflowd 0.9.8 installed from ports on FreeBSD 7.2-RELEASE p3. It is a virtual machine sitting on VMWare ESX using the em device driver and works great. It collects like a fiend with no lag whatsoever. My problem is when I issue 'softflowctl statistics'. After having softflowd running for about 15 minutes and issuing 'softflowctl statistics' on my busier link, I get: softflowd[13049]: Accumulated statistics: Number of active flows: 184115 Packets processed: 27177224 Fragments: 83352 Ignored packets: 27 (27 non-IP, 0 too short) Flows expired: 289532 (0 forced) Flows exported: 485703 in 14536 packets (133766 failures) Packets received by libpcap: 27235906 Packets dropped by libpcap: 58531 Packets dropped by interface: 63 The issue is in the flows reported line, namely the number of failures. I'm not sure if it's a matter of the send buffer needing to be increased or, well, what else it could be. I've checked network and disk utilisation on this machine and the flow collector and all are well under capacity - if anything, they are minimal usage. All virtual NICs are gigabit and the physical structure underneath is 100Mb. I'm about to create dedicated virtual NICs for the flow export so data never leaves the vswitch but I'm wondering if any of you folks that are monitoring gigabit links have some advice on where I can look to find the source of the failures (aside from having my sensor/collector running on virtual machines)? Scanning the archives has shown some softflowctl export errors from a few years ago but I get nothing in syslog about errors... Thanks! kmw -- Whenever there is in any country, uncultivated lands and unemployed poor, it is clear that the laws of property have been so far extended as to violate natural right. The earth is given as a common stock for man to labor and live on. -- Thomas Jefferson, 1785 From list2009 at lunch.za.net Fri Sep 11 02:26:31 2009 From: list2009 at lunch.za.net (Andrew McGill) Date: Thu, 10 Sep 2009 18:26:31 +0200 Subject: [netflow-tools] failed exports In-Reply-To: <5d6848b00909091223g3aec6137q4f3f768845649aab@mail.gmail.com> References: <5d6848b00909091223g3aec6137q4f3f768845649aab@mail.gmail.com> Message-ID: <200909101826.32087.list2009@lunch.za.net> Hi Kevin, It looks like you can get failures exporting flows when send() returns an error. The errno value is not captured, so you'll have to figure it out. You didn't say where the flows are going, but most likely, the netflow receiver that you are sending to is not able to collect all of the data at the rate it receives it, and the kernel is friendly enough to let you know about some icmp that floated back towards the sender. You can also get errors if you are getting interrupted system calls -- softflowd does not retry send() if it is interrupted. Maybe VMware is doing something like this to you? You can probably diagnose what the actual error is with strace -esend `pidof softflowd` -- or maybe -esendto (I haven't tried this myself). If you see EINTR, it should be easy to fix the code. &:-) On Wednesday 09 September 2009 21:23:30 Kevin Wilcox wrote: > Hi everyone. I'm running softflowd 0.9.8 installed from ports on > FreeBSD 7.2-RELEASE p3. > > It is a virtual machine sitting on VMWare ESX using the em device > driver and works great. It collects like a fiend with no lag > whatsoever. > > My problem is when I issue 'softflowctl statistics'. After having > softflowd running for about 15 minutes and issuing 'softflowctl > statistics' on my busier link, I get: > > softflowd[13049]: Accumulated statistics: > Number of active flows: 184115 > Packets processed: 27177224 > Fragments: 83352 > Ignored packets: 27 (27 non-IP, 0 too short) > Flows expired: 289532 (0 forced) > Flows exported: 485703 in 14536 packets (133766 failures) > Packets received by libpcap: 27235906 > Packets dropped by libpcap: 58531 > Packets dropped by interface: 63 > > The issue is in the flows reported line, namely the number of > failures. I'm not sure if it's a matter of the send buffer needing to > be increased or, well, what else it could be. I've checked network and > disk utilisation on this machine and the flow collector and all are > well under capacity - if anything, they are minimal usage. All virtual > NICs are gigabit and the physical structure underneath is 100Mb. I'm > about to create dedicated virtual NICs for the flow export so data > never leaves the vswitch but I'm wondering if any of you folks that > are monitoring gigabit links have some advice on where I can look to > find the source of the failures (aside from having my sensor/collector > running on virtual machines)? Scanning the archives has shown some > softflowctl export errors from a few years ago but I get nothing in > syslog about errors... > > Thanks! > > kmw From filip.palian at expro.pl Tue Sep 29 20:16:20 2009 From: filip.palian at expro.pl (Filip Palian) Date: Tue, 29 Sep 2009 12:16:20 +0200 Subject: [netflow-tools] Flowd + netflowdashboard + OpenBSD. Message-ID: <4AC1DE74.9010700@expro.pl> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi guys, Recently I run into some problems setting up flowd + netflowdashboard + OpenBSD. Luckily I found on google this thread - http://openbsd.monkey.org/ports/200505/msg00111.html. Apllying Damien's patch solved the problem. My question is, is it possible to merge this patch into the flowd sources? It may save some people a lot of time. - -- Best regards, Filip Palian -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQEcBAEBAgAGBQJKwd50AAoJEN++eipU1d8LjTMH/R2ZRxABVlGmjfVa5f91yPD9 uzC6MtlkSuOjaO6pK/xTL+n6WLRPI81NbQeQBPbAUFom+Z9RnIw+/F0zyqfPefFt dGZR8HbOJ/mZ6rqywOd7zCm26S6kuZEd27Ciytz+7zJj9ydocnKogTcXzohUaZWn /5zDWYtSsr1evXJcZ0DbE1i9ILyiE3WV8UFf1BV5xGzRQG9ApY9Nl97te7LcsZDk M8V/y+YBFir7BcohgBfu514KULj/jft1/FT3TvBRp7qcwcYoPXlupNvAZnqrF3IJ hNePaEOhxptvcpc2LdLG4GXO5K+zLWNgs8RH57B5kFdjxdMZa9uKATCOptJp1MQ= =qm98 -----END PGP SIGNATURE-----