From djm at mindrot.org Mon Apr 25 17:48:48 2005 From: djm at mindrot.org (Damien Miller) Date: Mon, 25 Apr 2005 17:48:48 +1000 Subject: [netflow-tools] Announce: py-radix-0.4 Message-ID: <426CA0E0.30601@mindrot.org> Hi, There is a new release (0.4) of py-radix out. This fixes a dumb bug that would corrupt stored prefixes with certain masklengths, but there are no other changes. I recommend that anyone using an older version of py-radix upgrade to this version. -d From jpietsch at gmail.com Thu Apr 28 10:13:55 2005 From: jpietsch at gmail.com (Justin Pietsch) Date: Wed, 27 Apr 2005 17:13:55 -0700 Subject: [netflow-tools] Why flowd Message-ID: <544b357705042717131d147d74@mail.gmail.com> Hi I've just discovered flowd and I have to ask why does flowd exist when there already is flow-tools. What does it do that cannot be done or is hard with flow-tools? thanks Justin From djm at mindrot.org Thu Apr 28 10:22:09 2005 From: djm at mindrot.org (Damien Miller) Date: Thu, 28 Apr 2005 10:22:09 +1000 Subject: [netflow-tools] Why flowd In-Reply-To: <544b357705042717131d147d74@mail.gmail.com> References: <544b357705042717131d147d74@mail.gmail.com> Message-ID: <42702CB1.50003@mindrot.org> Justin Pietsch wrote: > Hi > > I've just discovered flowd and I have to ask why does flowd exist when > there already is flow-tools. What does it do that cannot be done or is > hard with flow-tools? I haven't used flow-tools, I wrote flowd because I needed some particular features for a client (in particular, the fast front-end filtering that flowd does). Fortunately I was able to release it under a free license. -d From b.ghita at jack.see.plymouth.ac.uk Fri Apr 29 21:11:21 2005 From: b.ghita at jack.see.plymouth.ac.uk (Bogdan Ghita) Date: Fri, 29 Apr 2005 12:11:21 +0100 Subject: [netflow-tools] Softflowd and rrd running on the same machine Message-ID: <007801c54cac$2cf951f0$8360a38d@tweak> Hello everybody I've installed a few days ago softflowd and rrd-associated tools (flow-capture/flowscan) on a monitoring machine connected to a local network. First, I'll start with the praise - it's a great software, I've been looking for a long time at netflow-related software, but couldn't find anything that would allow me to implement it via a monitoring machine. Everything seems to be working reasonably well at the moment, but I still have a couple of problems that I can't find the solution for: - out-of-order packets. In order to get softflowd and rrd to work, I'm sending packets via the local interface of the machine. I thought this would work just fine but flow-capture reports continually 'lost' packets: Apr 29 11:27:18 linux flow-capture[23080]: ftpdu_seq_check(): src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 expecting=28199800 received=28199770 lost=4294967265 Apr 29 11:27:18 linux flow-capture[23080]: ftpdu_seq_check(): src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 expecting=28199800 received=28199770 lost=4294967265 Apr 29 11:27:18 linux flow-capture[23080]: ftpdu_seq_check(): src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 expecting=28199790 received=28199934 lost=144 Apr 29 11:27:18 linux flow-capture[23080]: ftpdu_seq_check(): src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 expecting=28199935 received=28199936 lost=1 Apr 29 11:27:19 linux flow-capture[23080]: ftpdu_seq_check(): src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 expecting=28199969 received=28199940 lost=4294967266 Apr 29 11:27:19 linux last message repeated 2 times Apr 29 11:27:19 linux flow-capture[23080]: ftpdu_seq_check(): src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 expecting=28199944 received=28200056 lost=112 Apr 29 11:27:20 linux flow-capture[23080]: ftpdu_seq_check(): src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 expecting=28200093 received=28200064 lost=4294967266 Apr 29 11:27:20 linux last message repeated 2 times Apr 29 11:27:20 linux flow-capture[23080]: ftpdu_seq_check(): src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 expecting=28200068 received=28200174 lost=106 Is it possible (I've don very little in terms of socket programming, so the answer might be straightforward) to pipe softflowd straight into flow-capture, so that the communication will be (hopefully) smoother? - CPU usage - softflowd seems to behave very strange when changing the priority - with the default priority (0), it uses continuously about 80% of the processor (yes, it is a big network and, again, yes, it is a slow machine - P3-900MHz); when I renice it (via 'top') to lower priority (e.g. 10), the utilisation drops to about 20%, although the processor is, overall, only about 50% used. Is the collector dropping packets/flow during that time? Is the machine that slow? - Reports - I am not sure whether this is a softflowd problem or an rrd-related problem. I've noticed a continuous udp flow running over the network, quite considerable in terms of bandwidth. However, when drawing the graphs, there is only an hourly spike, and nothing else. What could be causing this type of reporting? Thank you very much for your time. Regards Bogdan From djm at mindrot.org Sat Apr 30 09:11:49 2005 From: djm at mindrot.org (Damien Miller) Date: Sat, 30 Apr 2005 09:11:49 +1000 Subject: [netflow-tools] Softflowd and rrd running on the same machine In-Reply-To: <007801c54cac$2cf951f0$8360a38d@tweak> References: <007801c54cac$2cf951f0$8360a38d@tweak> Message-ID: <4272BF35.7060902@mindrot.org> Bogdan Ghita wrote: > Hello everybody > > I've installed a few days ago softflowd and rrd-associated tools > (flow-capture/flowscan) on a monitoring machine connected to a local > network. First, I'll start with the praise - it's a great software, I've > been looking for a long time at netflow-related software, but couldn't > find anything that would allow me to implement it via a monitoring > machine. Everything seems to be working reasonably well at the moment, > but I still have a couple of problems that I can't find the solution > for: > > - out-of-order packets. In order to get softflowd and rrd to work, I'm > sending packets via the local interface of the machine. I thought this > would work just fine but flow-capture reports continually 'lost' > packets: > > Apr 29 11:27:18 linux flow-capture[23080]: ftpdu_seq_check(): > src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 > expecting=28199800 received=28199770 lost=4294967265 > Apr 29 11:27:18 linux flow-capture[23080]: ftpdu_seq_check(): > src_ip=xxx.xxx.xxx.xxx dst_ip=xxx.xxx.xxx.xxx d_version=5 > expecting=28199800 received=28199770 lost=4294967265 This might be a bug in the sequence number generation of softflowd, or a bug in flow-capture - flow-capture is certainly printing negative sequence number offsets incorrectly. Can you capture some softflowd output packages with another netflow capable tool to manually check the sequence numbers? E.g. tcpdump's cnfp mode, flowd, etc. > Is it possible (I've don very little in terms of socket programming, so > the answer might be straightforward) to pipe softflowd straight into > flow-capture, so that the communication will be (hopefully) smoother? It is highly unlikely that the packets are acually getting dropped. It is more likely a bug in one/both packages. > - CPU usage - softflowd seems to behave very strange when changing the > priority - with the default priority (0), it uses continuously about 80% > of the processor (yes, it is a big network and, again, yes, it is a slow > machine - P3-900MHz); when I renice it (via 'top') to lower priority > (e.g. 10), the utilisation drops to about 20%, although the processor > is, overall, only about 50% used. Is the collector dropping packets/flow > during that time? Is the machine that slow? It is difficult to tell whether or not you are dropping packets - it depend a lot on what happens on the kernel side too. I'm not sure what information on packet drops libpcap makes available, but the attached patch adds printing of the statistics that it does provide to those printable using "softflowctl statistics". I'm not sure what a "dropped packet" in libpcap's parlance is - it may be a drop because the client couldn't keep up, or a drop from a bpf filter program. It should be obvious once you start playing with it though :) > - Reports - I am not sure whether this is a softflowd problem or an > rrd-related problem. I've noticed a continuous udp flow running over the > network, quite considerable in terms of bandwidth. However, when drawing > the graphs, there is only an hourly spike, and nothing else. What could > be causing this type of reporting? That depends on how the data is being presented - if you are showing flow records vs time, then a long UDP conversation may be represented by only a single flow record. Retrospectively producing throughput vs time representations of long-lived flow conversations is a little tricky and requires some suppositions, as the flow record summarises away any dynamics in the conversation. -d -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: softflowd-pcap-stats.diff Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050430/05f7807f/attachment.ksh From mwlucas at blackhelicopters.org Sat Apr 30 21:47:44 2005 From: mwlucas at blackhelicopters.org (Michael W. Lucas) Date: Sat, 30 Apr 2005 07:47:44 -0400 Subject: [netflow-tools] problems with pfflowd that don't happen with softflowd Message-ID: <20050430114744.GC60645@bewilderbeast.blackhelicopters.org> Hi, I'm currently using softflowd on FreeBSD 5.4, trying to switch over to pfflowd to get more timely exports of flows. (It *seems* that softflowd exports flows much later than when the traffic actually stops, and it *appears* that pfflowd expires these flows more quickly.) My collector is flow-capture, and works perfectly with softflowd. It doesn't actually record anything with pfflowd, however. If I run pfflowd in debug mode, it sure looks like I'm getting flows. ... pfflowd[40500]: FLOW proto 6 direction 1 pfflowd[40500]: start 2005-04-30T07:33:36(0) finish 2005-04-30T07:33:42(6880) pfflowd[40500]: w.x.y.z:10260 -> a.b.c.d:443 2897 bytes 11 packets pfflowd[40500]: a.b.c.d:443 -> w.x.y.z:10260 831 bytes 9 packets pfflowd[40500]: Sending flow packet len = 600 pfflowd[40500]: flows_exported = 36 ... tcpdump on the sensor and the collector shows that traffic is actually reaching the collector, so I don't think I've made an error on my host or port config. Anyone else seen this problem? I'm running pfflowd as: pfflowd -n a.b.c.d:port and softflowd as: softflowd -i em0 -n a.b.c.d:port Thanks for any suggestions! ==ml -- Michael W. Lucas mwlucas at FreeBSD.org, mwlucas at BlackHelicopters.org http://www.BlackHelicopters.org/~mwlucas/ Latest book: Cisco Routers for the Desperate http://www.CiscoRoutersForTheDesperate.com From djm at mindrot.org Sat Apr 30 21:58:30 2005 From: djm at mindrot.org (Damien Miller) Date: Sat, 30 Apr 2005 21:58:30 +1000 Subject: [netflow-tools] problems with pfflowd that don't happen with softflowd In-Reply-To: <20050430114744.GC60645@bewilderbeast.blackhelicopters.org> References: <20050430114744.GC60645@bewilderbeast.blackhelicopters.org> Message-ID: <427372E6.2090005@mindrot.org> Michael W. Lucas wrote: > Hi, > > I'm currently using softflowd on FreeBSD 5.4, trying to switch over to > pfflowd to get more timely exports of flows. (It *seems* that > softflowd exports flows much later than when the traffic actually > stops, and it *appears* that pfflowd expires these flows more > quickly.) That is possible: softflowd's timeouts are pretty conservative, especially for TCP - 30 minutes post-FIN. You can tune these on the commandline though :) > My collector is flow-capture, and works perfectly with softflowd. It > doesn't actually record anything with pfflowd, however. > > If I run pfflowd in debug mode, it sure looks like I'm getting flows. > > ... > pfflowd[40500]: FLOW proto 6 direction 1 > pfflowd[40500]: start 2005-04-30T07:33:36(0) finish 2005-04-30T07:33:42(6880) > pfflowd[40500]: w.x.y.z:10260 -> a.b.c.d:443 2897 bytes 11 packets > pfflowd[40500]: a.b.c.d:443 -> w.x.y.z:10260 831 bytes 9 packets > pfflowd[40500]: Sending flow packet len = 600 > pfflowd[40500]: flows_exported = 36 > ... > > tcpdump on the sensor and the collector shows that traffic is actually > reaching the collector, so I don't think I've made an error on my host > or port config. When you run tcpdump, did you try the "-T cnfp" to get it to parse the NetFlow packets? What collector are you using? -d From mwlucas at blackhelicopters.org Sat Apr 30 22:15:57 2005 From: mwlucas at blackhelicopters.org (Michael W. Lucas) Date: Sat, 30 Apr 2005 08:15:57 -0400 Subject: [netflow-tools] problems with pfflowd that don't happen with softflowd In-Reply-To: <427372E6.2090005@mindrot.org> References: <20050430114744.GC60645@bewilderbeast.blackhelicopters.org> <427372E6.2090005@mindrot.org> Message-ID: <20050430121557.GD60645@bewilderbeast.blackhelicopters.org> On Sat, Apr 30, 2005 at 09:58:30PM +1000, Damien Miller wrote: > Michael W. Lucas wrote: > >Hi, > > > >I'm currently using softflowd on FreeBSD 5.4, trying to switch over to > >pfflowd to get more timely exports of flows. (It *seems* that > >softflowd exports flows much later than when the traffic actually > >stops, and it *appears* that pfflowd expires these flows more > >quickly.) > > That is possible: softflowd's timeouts are pretty conservative, > especially for TCP - 30 minutes post-FIN. You can tune these on the > commandline though :) Ah, you learn something every day. That 30 minutes can be important when you want to, say, grovel through netflow data looking for port scans and worms and whatnot. :-) > When you run tcpdump, did you try the "-T cnfp" to get it to parse the > NetFlow packets? What collector are you using? I'm using flow-tools' flow-capture. Under softflowd, -T cnfp on the collector shows: 08:08:01.118622 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007 uptime, 1114862881.120821000, #0, 29 recs 08:08:01.118770 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007 uptime, 1114862881.120821000, #0, 30 recs 08:08:01.118918 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007 uptime, 1114862881.120821000, #0, 29 recs 08:08:01.118923 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007 uptime, 1114862881.120821000, #0, 29 recs with pfflowd, we see: 08:10:20.540514 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime, 1114863020.542900000, #0, 12 recs 08:10:20.540519 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime, 1114863020.542953000, #12, 12 recs 08:10:20.540662 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime, 1114863020.542971000, #24, 12 recs 08:10:20.540668 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime, 1114863020.542987000, #36, 11 recs The only difference I see is that with softflowd, the # doesn't increment. I'm willing to accept that there's something in flow-capture that's choking on something with pfflowd. Still, it seems that there will be more people on this list using flow-capture than there will be on the flow-capture list using pfflowd and softflowd. :-) I'm also using the same flow-capture binary to collect netflow from a Cisco router, so I somehow think that there's some detail with pfflowd that I just don't have right. ==ml -- Michael W. Lucas mwlucas at FreeBSD.org, mwlucas at BlackHelicopters.org http://www.BlackHelicopters.org/~mwlucas/ Latest book: Cisco Routers for the Desperate http://www.CiscoRoutersForTheDesperate.com From djm at mindrot.org Sat Apr 30 23:02:18 2005 From: djm at mindrot.org (Damien Miller) Date: Sat, 30 Apr 2005 23:02:18 +1000 Subject: [netflow-tools] problems with pfflowd that don't happen with softflowd In-Reply-To: <20050430121557.GD60645@bewilderbeast.blackhelicopters.org> References: <20050430114744.GC60645@bewilderbeast.blackhelicopters.org> <427372E6.2090005@mindrot.org> <20050430121557.GD60645@bewilderbeast.blackhelicopters.org> Message-ID: <427381DA.3000706@mindrot.org> Michael W. Lucas wrote: >>That is possible: softflowd's timeouts are pretty conservative, >>especially for TCP - 30 minutes post-FIN. You can tune these on the >>commandline though :) > > > Ah, you learn something every day. > > That 30 minutes can be important when you want to, say, grovel through > netflow data looking for port scans and worms and whatnot. :-) gah, it it not 30 minutes, it is really 5 minutes post-FIN. I probably started out writing 300 seconds, but botched changing the units. My apologies. >>When you run tcpdump, did you try the "-T cnfp" to get it to parse the >>NetFlow packets? What collector are you using? > > I'm using flow-tools' flow-capture. > > Under softflowd, -T cnfp on the collector shows: > > 08:08:01.118622 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007 uptime, 1114862881.120821000, #0, 29 recs > 08:08:01.118770 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007 uptime, 1114862881.120821000, #0, 30 recs > 08:08:01.118918 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007 uptime, 1114862881.120821000, #0, 29 recs > 08:08:01.118923 IP a.b.c.d.52011 > w.x.y.z.9318: NetFlow v5, 179.007 uptime, 1114862881.120821000, #0, 29 recs OK, that looks like a softflowd bug there - it is not updating the sequence number correctly. I can see one bug there (fix attached), but I don't see how it could give rise to all the sequence numbers being zero like this... > with pfflowd, we see: > > 08:10:20.540514 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime, 1114863020.542900000, #0, 12 recs > 08:10:20.540519 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime, 1114863020.542953000, #12, 12 recs > 08:10:20.540662 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime, 1114863020.542971000, #24, 12 recs > 08:10:20.540668 IP a.b.c.d.57523 > w.x.y.z.9318: NetFlow v5, 8.741 uptime, 1114863020.542987000, #36, 11 recs > > The only difference I see is that with softflowd, the # doesn't > increment. There might be more to it. Could you try capturing with "tcpdump -s1500 -vvvTcnfp"? It will show some more details. > I'm willing to accept that there's something in flow-capture that's > choking on something with pfflowd. Still, it seems that there will be > more people on this list using flow-capture than there will be on the > flow-capture list using pfflowd and softflowd. :-) > > I'm also using the same flow-capture binary to collect netflow from a > Cisco router, so I somehow think that there's some detail with pfflowd > that I just don't have right. pfflowd doesn't fill in all the NetFlow fields - it just doesn't get enough information for the pfsync interface to do it. IIRC some programs may not like getting a 0 interface index. You could try the second attached patch, which is a very crude hack that might fool it (just hardcode a couple of interface numbers). -d -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: softflowd-seqno.diff Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050430/3d06409c/attachment.ksh -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pfflowd-fake-ifndx.diff Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050430/3d06409c/attachment-0001.ksh