From djm at mindrot.org Sun Sep 4 19:59:59 2005 From: djm at mindrot.org (Damien Miller) Date: Sun, 04 Sep 2005 19:59:59 +1000 Subject: [netflow-tools] Call for testing Message-ID: <431AC59F.70201@mindrot.org> Hi, I'd like to do a new flowd release soon. Before this happens, it would be good to get some test reports on various systems. I have the ability to test on a OpenBSD, a couple of Linux variants and Solaris 9/sparc, but this isn't as wide a field as I would like. This also applies to testing flowd with as many NetFlow source devices as possible. Apart from softflowd and pfflowd, presently I only have access to couple of old Cisco routers as sources (none of which can generate NetFlow v.9). It would be good if readers of this list could try one of the snapshot releases at http://www2.mindrot.org/flowd_snap/ and report back with details of their system. In particular, tests of the new Python API are welcome. Thanks, Damien Miller From Maiko.Wessel at smail.inf.fh-bonn-rhein-sieg.de Tue Sep 6 07:09:54 2005 From: Maiko.Wessel at smail.inf.fh-bonn-rhein-sieg.de (Maiko Wessel) Date: Mon, 05 Sep 2005 23:09:54 +0200 Subject: [netflow-tools] softflowd TOS Message-ID: <431CB422.7000202@smail.inf.fh-bonn-rhein-sieg.de> Hi, is there a patch to log the tos field from ipv4/ipv4 packets with the softflowd? We use softflowd and flowd and we need the bits from tos field. I've looked at the flowd source and they are ready to read the tos value from the netflow packets, but softflowd can't emit netflow packets with tos?!? From djm at mindrot.org Tue Sep 6 08:01:30 2005 From: djm at mindrot.org (Damien Miller) Date: Tue, 06 Sep 2005 08:01:30 +1000 Subject: [netflow-tools] softflowd TOS In-Reply-To: <431CB422.7000202@smail.inf.fh-bonn-rhein-sieg.de> References: <431CB422.7000202@smail.inf.fh-bonn-rhein-sieg.de> Message-ID: <431CC03A.50003@mindrot.org> Maiko Wessel wrote: > Hi, > > is there a patch to log the tos field from ipv4/ipv4 packets with the > softflowd? > > We use softflowd and flowd and we need the bits from tos field. I've > looked at the flowd source > and they are ready to read the tos value from the netflow packets, but > softflowd can't emit netflow > packets with tos?!? Please try the attached patch. It tries to learn the ToS from the first packet in the flow only, as this is what usually determines how the flow is treated by routers. Warning: I wrote this in a hurry - please test that NetFlow v.9 works in particular Regards, Damien Miller -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: softflowd-tos.diff Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050906/32d21d0d/attachment.ksh From dm at mainframe.ca Wed Sep 14 09:33:32 2005 From: dm at mainframe.ca (Derrick MacPherson) Date: Tue, 13 Sep 2005 16:33:32 -0700 Subject: [netflow-tools] Hi. Basic knowledge sought. Message-ID: <1126654412.8813.66.camel@Mandarin-04.mainframe.ca> I have this scenario: Internet | PIX / | \ VPN LAN DMZ I want to gather stats, (total sent/rec'd, break down of types of traffic, etc) and am mainly concerned with the LAN traffic, if that effects any suggestions. I've been floundering around trying to find some answers, and wasting a lot of time. I just found out that the PIX tell you crap from snmp, so setting up cricket has amounted in nothing of any use. I have a machine that I can insert as a bridge tween the LAN and the Pix, or on the other side of the Pix. Can someone suggest a direction/solution? Thanks. D From gijs at looze.net Fri Sep 16 21:44:21 2005 From: gijs at looze.net (Gijs Molenaar) Date: Fri, 16 Sep 2005 13:44:21 +0200 Subject: [netflow-tools] what happens with lost flows? Message-ID: <432AB015.5050606@looze.net> Hello, What will happen when the system load is very high, and flowd can't process incomming flows? will these flows be dropped? If they are dropped, is there a way to discover how many flows are dropped? greetings, gijs From djm at mindrot.org Sat Sep 17 10:23:06 2005 From: djm at mindrot.org (Damien Miller) Date: Sat, 17 Sep 2005 10:23:06 +1000 Subject: [netflow-tools] what happens with lost flows? In-Reply-To: <432AB015.5050606@looze.net> References: <432AB015.5050606@looze.net> Message-ID: <432B61EA.1080102@mindrot.org> Gijs Molenaar wrote: > Hello, > > What will happen when the system load is very high, and flowd can't > process incomming flows? will these flows be dropped? If they are > dropped, is there a way to discover how many flows are dropped? There is a small kernel-maintained input queue where some will be stored if flowd can't keep up, after that they are dropped, but if the system load is too high for something light like flowd then this will fill up pretty quickly. You can figure out if there have been drops by looking at "flow_sequence" in the FLOW_ENGINE_INFO section (obviously you have to direct flowd to store it). -d From rbreathe at brookes.ac.uk Mon Sep 26 17:57:30 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Mon, 26 Sep 2005 08:57:30 +0100 Subject: [netflow-tools] Flow time query Message-ID: <4337A9EA.5000904@brookes.ac.uk> Greetings, I'm trying to work out whether flow-tools will allow me to retrieve (or calculate) a second-accurate flow start-time in seconds since the UNIX epoch. If my understanding is correct, and refering to the NetFlow v9 specification along with store.h, AGENT_INFO contains time_sec & time_nanosec, but these appear to always take the same value as RECV_TIME.recv_sec. I want to calculate a flows start and stop times relative to unix epoch rather than the devices uptime. Would the following give me what I'm looking for? actual_flows_start = (AGENT_INFO.time_sec - 100*AGENT_INFO.sys_uptime_ms) + FLOW_TIMES.flows_start Is there a more sane/sensible way? On a semi-related note, I've locally patched flowd-reader to support export to SQLite to facilitate further analysis. Would anyone else be interested in my cleaning up my patches and submitting them? Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050926/9051bdd9/attachment.bin From djm at mindrot.org Mon Sep 26 18:55:04 2005 From: djm at mindrot.org (Damien Miller) Date: Mon, 26 Sep 2005 18:55:04 +1000 Subject: [netflow-tools] Flow time query In-Reply-To: <4337A9EA.5000904@brookes.ac.uk> References: <4337A9EA.5000904@brookes.ac.uk> Message-ID: <4337B768.70409@mindrot.org> Robin Breathe wrote: > Greetings, > > I'm trying to work out whether flow-tools will allow me to retrieve (or > calculate) a second-accurate flow start-time in seconds since the UNIX > epoch. > > If my understanding is correct, and refering to the NetFlow v9 > specification along with store.h, AGENT_INFO contains time_sec & > time_nanosec, but these appear to always take the same value as > RECV_TIME.recv_sec. They are in fact different: recv_sec is set by flowd, but time_sec is set by the flow exported. If they are the same, then it is because your clocks are in sync and there was no lag between flow expiry and export. On a Cisco, for example, the first flows in an export packet usually have the largest delta between recv_sec, as they were expired and subsequently queued first. On a busy router, there may not be much of a lag though. On the other hand, if you are using something like softflowd that exports flows as soon as they expire, then the only difference that you will see is the difference between the exporter and the collector's clock, or zero if they are on the same host. > I want to calculate a flows start and stop times > relative to unix epoch rather than the devices uptime. > > Would the following give me what I'm looking for? > > actual_flows_start = > (AGENT_INFO.time_sec - 100*AGENT_INFO.sys_uptime_ms) > + FLOW_TIMES.flows_start Shouldn't this be: time_sec - (sys_uptime_ms / 1000) + flow_start ? > Is there a more sane/sensible way? I don't think so. BTW this is already in the TODO list, so it would be a welcome addition even if it just a helper macro or two in store.h. > On a semi-related note, I've locally patched flowd-reader to support > export to SQLite to facilitate further analysis. Would anyone else be > interested in my cleaning up my patches and submitting them? Probably not as part of flowd-reader, but a separate tool (to live in the tools/ subdirectory) would be most welcome. There is already a Perl script to do just this there. -d From rbreathe at brookes.ac.uk Mon Sep 26 19:26:45 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Mon, 26 Sep 2005 10:26:45 +0100 Subject: [netflow-tools] Flow time query In-Reply-To: <4337A9EA.5000904@brookes.ac.uk> References: <4337A9EA.5000904@brookes.ac.uk> Message-ID: <4337BED5.4070104@brookes.ac.uk> The actual_flows_start equation I gave before was total junk. Does the following look more reasonable? unix_flow_start = (ntohl(flow->ainfo.time_sec) - ntohl(flow->ainfo.sys_uptime_ms)/1000) + ntohl(flow->ftimes.flow_start)/1000; This appears to give relatively sensible results, but is it correct? Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050926/37c5f282/attachment.bin From rbreathe at brookes.ac.uk Mon Sep 26 19:41:10 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Mon, 26 Sep 2005 10:41:10 +0100 Subject: [netflow-tools] Flow time query In-Reply-To: <4337B768.70409@mindrot.org> References: <4337A9EA.5000904@brookes.ac.uk> <4337B768.70409@mindrot.org> Message-ID: <4337C236.4050403@brookes.ac.uk> Damien Miller wrote: > ...snip... > Shouldn't this be: > > time_sec - (sys_uptime_ms / 1000) + flow_start > > ? Pretty much (see my other post). >> Is there a more sane/sensible way? > > I don't think so. > > BTW this is already in the TODO list, so it would be a welcome addition > even if it just a helper macro or two in store.h. I'll have a look. >> On a semi-related note, I've locally patched flowd-reader to support >> export to SQLite to facilitate further analysis. Would anyone else be >> interested in my cleaning up my patches and submitting them? > > Probably not as part of flowd-reader, but a separate tool (to live in > the tools/ subdirectory) would be most welcome. There is already a > Perl script to do just this there. Yup, I had trouble getting the perl module to work under solaris9/sparc, so decided to patch it into flowd-reader (since I could then also exploit its existing filter code). A feature of my SQLite exporter is that it will merge both multiple instances of a flow and the two sides of a bidirectional dataflow into a single row (by grouping on the src_addr:src_port/dst_addr:dst_port/proto tuple). I have another utility to merge flows in adjacent exports into a single table. However, at present both of these utilities only work on a fixed set of columns: start_time, duration, {src,dst}_{addr,port}, proto, {in,out}_{octets,packets} Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050926/af9b08c3/attachment.bin From rbreathe at brookes.ac.uk Mon Sep 26 22:41:50 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Mon, 26 Sep 2005 13:41:50 +0100 Subject: [netflow-tools] softflowctl expire-all Message-ID: <4337EC8E.8000400@brookes.ac.uk> I use softflowd and flowd together on a Solaris 9 host talking NetFlow v5. I seem to be seeing an inconsistency between the output of `softflowctl statistics` and the results of issuing a `softflowctl expire-all`. I expect that issuing an `expire-all` would force softflowd to export all of its current flow data to flowd and restart monitoring. However, running a `flowd-reader -v flows.db | wc -l` before and after indicates that this is not the case. Example output: ##### BEGIN # softflowctl statistics; \ > echo "%%%flows: `flowd-reader -v flows.raw | wc -l`"; \ > softflowctl expire-all; \ > echo "%%%flows: `flowd-reader -v flows.raw | wc -l`"; \ > softflowctl statistics statistics softflowd[11574]: Accumulated statistics: Number of active flows: 4176 Packets processed: 9372374 Fragments: 2 Ignored packets: 1405 (1405 non-IP, 0 too short) Flows expired: 119941 (0 forced) Flows exported: 239882 in 7569 packets (0 failures) Expired flow statistics: minimum average maximum Flow bytes: 46 51562 79795808 Flow packets: 1 76 109762 Duration: 0.00s 18.83s 299.70s Expired flow reasons: tcp = 0 tcp.rst = 5895 tcp.fin = 0 udp = 0 icmp = 0 general = 0 maxlife = 0 over 2Gb = 0 maxflows = 0 flushed = 114046 Per-protocol statistics: Octets Packets Avg Life Max Life Unknown (1): 162797 2397 19.20s 298.84s Unknown (6): 5939745100 8157978 18.67s 299.70s Unknown (17): 244516151 895628 19.53s 299.69s Unknown (41): 1088 16 2.93s 6.06s %%%flows 354 expire-all softflowd[11574]: Expired 4181 flows. %%%flows 531 statistics softflowd[11574]: Accumulated statistics: Number of active flows: 0 Packets processed: 9372970 Fragments: 2 Ignored packets: 1405 (1405 non-IP, 0 too short) Flows expired: 124122 (0 forced) Flows exported: 248244 in 7833 packets (0 failures) Expired flow statistics: minimum average maximum Flow bytes: 46 51506 79795808 Flow packets: 1 76 109762 Duration: 0.00s 18.54s 299.70s Expired flow reasons: tcp = 0 tcp.rst = 5895 tcp.fin = 0 udp = 0 icmp = 0 general = 0 maxlife = 0 over 2Gb = 0 maxflows = 0 flushed = 118227 Per-protocol statistics: Octets Packets Avg Life Max Life Unknown (1): 167651 2469 19.32s 298.84s Unknown (6): 6134757449 8419521 18.33s 299.70s Unknown (17): 258098658 950964 19.46s 299.69s Unknown (41): 1088 16 2.93s 6.06s ##### END It seems as though "flows expired" is increasing by about the right amount, "flows exported" is going up by a factor of two over the number of active flows, and the flowd datafile is barely going up at all! # cat flowd.conf listen on 127.0.0.1:4432 flow source 127.0.0.1 store ALL logfile "/data/netflow/flows.raw" Is my understanding of the way netflow, and in particular netflow-tools, works flawed? Any ideas on how to proceed in working out what's going wrong? Regards, Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050926/10225e45/attachment.bin From djm at mindrot.org Tue Sep 27 08:00:15 2005 From: djm at mindrot.org (Damien Miller) Date: Tue, 27 Sep 2005 08:00:15 +1000 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <4337EC8E.8000400@brookes.ac.uk> References: <4337EC8E.8000400@brookes.ac.uk> Message-ID: <43386F6F.6010205@mindrot.org> Robin Breathe wrote: > I use softflowd and flowd together on a Solaris 9 host talking NetFlow v5. > > I seem to be seeing an inconsistency between the output of `softflowctl > statistics` and the results of issuing a `softflowctl expire-all`. hm, this is weird... > Is my understanding of the way netflow, and in particular netflow-tools, > works flawed? Any ideas on how to proceed in working out what's going wrong? Could you try running tcpdump to capture the export packets to see which number is correct? -d From rbreathe at brookes.ac.uk Tue Sep 27 09:25:41 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Tue, 27 Sep 2005 00:25:41 +0100 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <43386F6F.6010205@mindrot.org> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> Message-ID: <43388375.80801@brookes.ac.uk> Damien Miller wrote: > Robin Breathe wrote: >> I use softflowd and flowd together on a Solaris 9 host talking NetFlow >> v5. >> >> I seem to be seeing an inconsistency between the output of `softflowctl >> statistics` and the results of issuing a `softflowctl expire-all`. > > hm, this is weird... > >> Is my understanding of the way netflow, and in particular netflow-tools, >> works flawed? Any ideas on how to proceed in working out what's going >> wrong? > > Could you try running tcpdump to capture the export packets to see which > number is correct? Damien, I actually did some fairly extensive testing earlier, I simply couldn't decide how best to report the results. Running tcpdump shows that there really are far more active flows than end up in my flowd logs. Running both softflowd and flowd in debug mode, it becomes apparent that softflowd *thinks* that it is expiring all the flows (it prints the "EXPIRED:" line in check_expired()), but some never arrive with flowd. If I initiate a test flow (e.g. complete ssh session), then if I do an `expire-all` after a few seconds, it won't get exported; if I wait for it to naturally expire, it is exported. At least this is easily reproduceable :) The end result of this is that taking "snapshots" with `expire-all` every 5 minutes *usually* gives me only the first 3 minutes worth of flows (assuming the real flow-start-time algorithm we discussed earlier is sane). The exact period which gets exported seems to vary a little (+/-90s), though this is likely due to varying traffic patterns (running on a network with >1500 active hosts). Your help is really appreciated, this was an unexpected surprise drawing close to the end of an otherwise successful project :) I was going to trawl through the code myself tomorrow, but you already know it; I'd be even more delighted if we could nail this bug down together. Kind regards, Robin -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/baab7979/attachment.bin -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/baab7979/attachment-0001.bin From djm at mindrot.org Tue Sep 27 09:47:58 2005 From: djm at mindrot.org (Damien Miller) Date: Tue, 27 Sep 2005 09:47:58 +1000 (EST) Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <43388375.80801@brookes.ac.uk> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> Message-ID: On Tue, 27 Sep 2005, Robin Breathe wrote: > Damien, > > I actually did some fairly extensive testing earlier, I simply couldn't > decide how best to report the results. > > Running tcpdump shows that there really are far more active flows than > end up in my flowd logs. Are you tcpdumping the traffic flows themselves or the flow export packets? To ask an obvious question: are you using any flowd.conf filters that drop flows? > Your help is really appreciated, this was an unexpected surprise drawing > close to the end of an otherwise successful project :) I was going to > trawl through the code myself tomorrow, but you already know it; I'd be > even more delighted if we could nail this bug down together. I'm setting up a test environment right now. Hopefully this will be fairly easy to track down. (famous last words...) -d From rbreathe at brookes.ac.uk Tue Sep 27 16:54:08 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Tue, 27 Sep 2005 07:54:08 +0100 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> Message-ID: <4338EC90.6090907@brookes.ac.uk> Damien Miller wrote: > On Tue, 27 Sep 2005, Robin Breathe wrote: >> I actually did some fairly extensive testing earlier, I simply couldn't >> decide how best to report the results. >> >> Running tcpdump shows that there really are far more active flows than >> end up in my flowd logs. > > Are you tcpdumping the traffic flows themselves or the flow export packets? On the traffic flows themselves; I'm running softflowd and flowd on the same Solaris 9 box, and S9 doesn't support tcpdump on the loopback interface. If necessary I can set the two components up on different machines allowing me to tcpdump the flow export packets... > To ask an obvious question: are you using any flowd.conf filters that > drop flows? I can replicate the issue on a vanilla configuration: listen on 127.0.0.1:4432 flow source 127.0.0.1 accept all store ALL logfile "/data/netflow/flows.raw" I have tried with and without bpf filters on softflowd: they make no difference (but did highlight another possible far-less-important bug to be looked at another time - not all valid recipes are accepted). >> Your help is really appreciated, this was an unexpected surprise drawing >> close to the end of an otherwise successful project :) I was going to >> trawl through the code myself tomorrow, but you already know it; I'd be >> even more delighted if we could nail this bug down together. > > I'm setting up a test environment right now. Hopefully this will be > fairly easy to track down. (famous last words...) Probably wrong, I've got it into my head that this relates to iterating over the expiries list rather than the flows list in the `expire-all` case. I'll look at the code once I get into work. Robin -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/d091a8da/attachment.bin From rbreathe at brookes.ac.uk Tue Sep 27 18:00:22 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Tue, 27 Sep 2005 09:00:22 +0100 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <4338EC90.6090907@brookes.ac.uk> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> Message-ID: <4338FC16.2040409@brookes.ac.uk> Robin Breathe wrote: > Probably wrong, I've got it into my head that this relates to iterating > over the expiries list rather than the flows list in the `expire-all` > case. I'll look at the code once I get into work. Having looked at the code, my head is cleansed of this idea :) Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/3ca532e7/attachment.bin From djm at mindrot.org Tue Sep 27 20:17:44 2005 From: djm at mindrot.org (Damien Miller) Date: Tue, 27 Sep 2005 20:17:44 +1000 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <43391AF9.7040807@brookes.ac.uk> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> <4338FC16.2040409@brookes.ac.uk> <43391AF9.7040807@brookes.ac.uk> Message-ID: <43391C48.5060708@mindrot.org> Robin Breathe wrote: > Damien, > > I've managed to replicate the problem running with a stock installation > of softflowd/flowd from ports under FreeBSD 6.0-BETA5. Yes, I have replicated it too. It looks like flowd can lose packets when they come in large sudden bursts. I'm adding an input buffer and changing the processing order to better cope with these situations. -d From rbreathe at brookes.ac.uk Tue Sep 27 20:28:03 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Tue, 27 Sep 2005 11:28:03 +0100 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <43391C48.5060708@mindrot.org> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> <4338FC16.2040409@brookes.ac.uk> <43391AF9.7040807@brookes.ac.uk> <43391C48.5060708@mindrot.org> Message-ID: <43391EB3.905@brookes.ac.uk> Damien Miller wrote: >> I've managed to replicate the problem running with a stock installation >> of softflowd/flowd from ports under FreeBSD 6.0-BETA5. > > Yes, I have replicated it too. It looks like flowd can lose packets when > they come in large sudden bursts. I'm adding an input buffer and > changing the processing order to better cope with these situations. Excellent! Let me know when it's ready to test :) Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/655936ef/attachment.bin From rbreathe at brookes.ac.uk Tue Sep 27 21:28:16 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Tue, 27 Sep 2005 12:28:16 +0100 Subject: [netflow-tools] softflowd statistics bug Message-ID: <43392CD0.1010903@brookes.ac.uk> Damien, In softflowd.c, it is falsely assumed that every expired flow will lead to 2 exports. This has made testing of the other bug (in flowd) a little harder as softflowd reports that it has exported more flows than it really has. Line 782: ft->flows_exported += num_expired * 2; Line 786: ft->flows_dropped += num_expired * 2; In the case of a unidirectional flow, this is not the case. The flows_exported case is relatively easy to fix, the flows_dropped case probably isn't worth it. I'm happy to work on a patch, but how would you rather I approach it? 1) Have send_netflow_*() return a new struct: { u_int32_t num_packets; uint32_t num_flows } 2) Add such a struct to struct FLOWTRACK and have send_netflow_*() initialise and update it on each expiry run. 3) ...something else? Regards, Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/acf4c96b/attachment.bin From djm at mindrot.org Tue Sep 27 22:30:58 2005 From: djm at mindrot.org (Damien Miller) Date: Tue, 27 Sep 2005 22:30:58 +1000 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <43391EB3.905@brookes.ac.uk> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> <4338FC16.2040409@brookes.ac.uk> <43391AF9.7040807@brookes.ac.uk> <43391C48.5060708@mindrot.org> <43391EB3.905@brookes.ac.uk> Message-ID: <43393B82.9020508@mindrot.org> Robin Breathe wrote: > Damien Miller wrote: > >>>I've managed to replicate the problem running with a stock installation >>>of softflowd/flowd from ports under FreeBSD 6.0-BETA5. >> >>Yes, I have replicated it too. It looks like flowd can lose packets when >>they come in large sudden bursts. I'm adding an input buffer and >>changing the processing order to better cope with these situations. > > > Excellent! Let me know when it's ready to test :) Here it is. The attached patch adds an input queue to flowd. When a listening fd becomes ready (i.e. flow data is received), flowd will read and enqueue up to 1024 packets before processing them. It won't stop situations where flowd simply can't keep up with the data coming at it (my laptop for example drops after ~1k flows per second), but it should help it deal better with sudden bursts of flows. I'll tweak this a bit further, I think it could be made a little faster by replacing the allocations of queued packets with a freelist of preallocated ones. It would probably be better to maintain an output queue as well, instead of doing one write per flow. This is against flowd -current, so if you aren't already using a snapshot, then you will need to get one from: http://www2.mindrot.org/flowd_snap/ -d -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: flowd-input-buffer.diff Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/0f50cf41/attachment.ksh From rbreathe at brookes.ac.uk Wed Sep 28 01:14:25 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Tue, 27 Sep 2005 16:14:25 +0100 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <43393B82.9020508@mindrot.org> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> <4338FC16.2040409@brookes.ac.uk> <43391AF9.7040807@brookes.ac.uk> <43391C48.5060708@mindrot.org> <43391EB3.905@brookes.ac.uk> <43393B82.9020508@mindrot.org> Message-ID: <433961D1.7040009@brookes.ac.uk> Damien Miller wrote: > Here it is. > > The attached patch adds an input queue to flowd. When a listening fd > becomes ready (i.e. flow data is received), flowd will read and enqueue > up to 1024 packets before processing them. > > It won't stop situations where flowd simply can't keep up with the data > coming at it (my laptop for example drops after ~1k flows per second), > but it should help it deal better with sudden bursts of flows. > > I'll tweak this a bit further, I think it could be made a little faster > by replacing the allocations of queued packets with a freelist of > preallocated ones. It would probably be better to maintain an output > queue as well, instead of doing one write per flow. > > This is against flowd -current, so if you aren't already using a > snapshot, then you will need to get one from: > > http://www2.mindrot.org/flowd_snap/ I'm beginning to test the 20050927 snapshot (which looks to have the queue patch), but needed to apply the attached patch in order for it to compile in Solaris 9. It also fails to configure if bison is present, is this intentional? Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.diff Url: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/3a9585bd/attachment.ksh -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050927/3a9585bd/attachment.bin From djm at mindrot.org Wed Sep 28 07:58:36 2005 From: djm at mindrot.org (Damien Miller) Date: Wed, 28 Sep 2005 07:58:36 +1000 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <433961D1.7040009@brookes.ac.uk> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> <4338FC16.2040409@brookes.ac.uk> <43391AF9.7040807@brookes.ac.uk> <43391C48.5060708@mindrot.org> <43391EB3.905@brookes.ac.uk> <43393B82.9020508@mindrot.org> <433961D1.7040009@brookes.ac.uk> Message-ID: <4339C08C.9020706@mindrot.org> Robin Breathe wrote: > I'm beginning to test the 20050927 snapshot (which looks to have the > queue patch), but needed to apply the attached patch in order for it to > compile in Solaris 9. No, the queue patch has not been committed. > It also fails to configure if bison is present, is this intentional? configure should use Berkeley yacc in favour of bison. Could you send me a compilation log of these two errors in action? I have compiled on Solaris 9 without problems recently. -d From djm at mindrot.org Wed Sep 28 10:02:01 2005 From: djm at mindrot.org (Damien Miller) Date: Wed, 28 Sep 2005 10:02:01 +1000 (EST) Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <4339C08C.9020706@mindrot.org> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> <4338FC16.2040409@brookes.ac.uk> <43391AF9.7040807@brookes.ac.uk> <43391C48.5060708@mindrot.org> <43391EB3.905@brookes.ac.uk> <43393B82.9020508@mindrot.org> <433961D1.7040009@brookes.ac.uk> <4339C08C.9020706@mindrot.org> Message-ID: Sorry for this terse response, I'll expand a little. On Wed, 28 Sep 2005, Damien Miller wrote: > Robin Breathe wrote: >> I'm beginning to test the 20050927 snapshot (which looks to have the >> queue patch), but needed to apply the attached patch in order for it to >> compile in Solaris 9. > > No, the queue patch has not been committed. You will have to apply it yourself. Please let me know how it goes and I'll commit it if it is stable and improves your situation. Note that the snapshot releases change the log format a bit from the last stable release. You can convert your logs using flowd-reader's -L option. >> It also fails to configure if bison is present, is this intentional? > > configure should use Berkeley yacc in favour of bison. GNU Bison is known to miscompile parse.y and I haven't bothered to spend the time to figure out why. On Solaris, /usr/ccs/bin/yacc is known to do the right thing (on Linux, use byacc). -d From djm at mindrot.org Wed Sep 28 10:36:22 2005 From: djm at mindrot.org (Damien Miller) Date: Wed, 28 Sep 2005 10:36:22 +1000 (EST) Subject: [netflow-tools] softflowd statistics bug In-Reply-To: <43392CD0.1010903@brookes.ac.uk> References: <43392CD0.1010903@brookes.ac.uk> Message-ID: On Tue, 27 Sep 2005, Robin Breathe wrote: > Damien, > > In softflowd.c, it is falsely assumed that every expired flow will lead > to 2 exports. > This has made testing of the other bug (in flowd) a little harder as > softflowd reports that it has exported more flows than it really has. > > Line 782: ft->flows_exported += num_expired * 2; > Line 786: ft->flows_dropped += num_expired * 2; > > In the case of a unidirectional flow, this is not the case. I think this is fixed in the development version, of which I have just set up public snapshot releases. Have a look at: http://www2.mindrot.org/softflowd_snap/ -d From rbreathe at brookes.ac.uk Wed Sep 28 18:17:55 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Wed, 28 Sep 2005 09:17:55 +0100 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> <4338FC16.2040409@brookes.ac.uk> <43391AF9.7040807@brookes.ac.uk> <43391C48.5060708@mindrot.org> <43391EB3.905@brookes.ac.uk> <43393B82.9020508@mindrot.org> <433961D1.7040009@brookes.ac.uk> <4339C08C.9020706@mindrot.org> Message-ID: <433A51B3.7060308@brookes.ac.uk> Damien Miller wrote: > On Wed, 28 Sep 2005, Damien Miller wrote: >> Robin Breathe wrote: >>> I'm beginning to test the 20050927 snapshot (which looks to have the >>> queue patch), but needed to apply the attached patch in order for it to >>> compile in Solaris 9. >> >> No, the queue patch has not been committed. > > You will have to apply it yourself. Please let me know how it goes and > I'll commit it if it is stable and improves your situation. The patch didn't apply cleanly as given (it didn't like the process_netflow_v7 function), but I munged it in. However, I still seem to be getting forgotten flows: ## logged 1 lines # sleep 30 # softflowctl stop-gather stop-gather softflowd[3271]: Data collection stopped. ## logged 1 lines # softflowctl statistics statistics softflowd[3271]: Accumulated statistics: Number of active flows: 1570 Packets processed: 101136 Fragments: 0 Ignored packets: 21 (21 non-IP, 0 too short) Flows expired: 0 (0 forced) Flows exported: 0 in 0 packets (0 failures) # softflowctl expire-all expire-all softflowd[3271]: Expired 1570 flows. # softflowctl statistics statistics softflowd[3271]: Accumulated statistics: Number of active flows: 0 Packets processed: 101136 Fragments: 0 Ignored packets: 21 (21 non-IP, 0 too short) Flows expired: 1570 (0 forced) Flows exported: 3140 in 98 packets (0 failures) Expired flow statistics: minimum average maximum Flow bytes: 46 48461 34269298 Flow packets: 1 64 34677 Duration: 0.00s 4.88s 31.27s Expired flow reasons: tcp = 0 tcp.rst = 0 tcp.fin = 0 udp = 0 icmp = 0 general = 0 maxlife = 0 over 2Gb = 0 maxflows = 0 flushed = 1570 Per-protocol statistics: Octets Packets Avg Life Max Life Unknown (1): 545 5 0.51s 2.05s Unknown (6): 73832643 92504 4.89s 31.26s Unknown (17): 2250079 8617 4.92s 31.27s Unknown (41): 680 10 4.40s 6.00s ## logged 176 lines In fact, if anything it seems to be worse? Should I try increasing INPUT_MAX_PACKET_PER_FD? > Note that the snapshot releases change the log format a bit from the > last stable release. You can convert your logs using flowd-reader's -L > option. Yup, this will force me to update my flowdb->sqlite conversion program, which is probably not a bad thing (I'll dis-entangle it from flowd-reader). >>> It also fails to configure if bison is present, is this intentional? >> >> configure should use Berkeley yacc in favour of bison. > > GNU Bison is known to miscompile parse.y and I haven't bothered to spend > the time to figure out why. On Solaris, /usr/ccs/bin/yacc is known to do > the right thing (on Linux, use byacc). Fair enough, just unexpected :) Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050928/4aabdd73/attachment.bin From rbreathe at brookes.ac.uk Wed Sep 28 19:43:11 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Wed, 28 Sep 2005 10:43:11 +0100 Subject: [netflow-tools] softflowctl expire-all In-Reply-To: <433A51B3.7060308@brookes.ac.uk> References: <4337EC8E.8000400@brookes.ac.uk> <43386F6F.6010205@mindrot.org> <43388375.80801@brookes.ac.uk> <4338EC90.6090907@brookes.ac.uk> <4338FC16.2040409@brookes.ac.uk> <43391AF9.7040807@brookes.ac.uk> <43391C48.5060708@mindrot.org> <43391EB3.905@brookes.ac.uk> <43393B82.9020508@mindrot.org> <433961D1.7040009@brookes.ac.uk> <4339C08C.9020706@mindrot.org> <433A51B3.7060308@brookes.ac.uk> Message-ID: <433A65AF.6000400@brookes.ac.uk> Damien, Below are the results of using flowd-SNAP-20050927 w/flowd-input-buffer.diff along with softflowd-SNAP-20050928 (for enhanced statistical accuracy): ## logged 1 lines # sleep 30 # softflowctl stop-gather stop-gather softflowd[16769]: Data collection stopped. ## logged 1 lines # softflowctl statistics statistics softflowd[16769]: Accumulated statistics: Number of active flows: 2770 Packets processed: 78026 Fragments: 0 Ignored packets: 23 (23 non-IP, 0 too short) Flows expired: 0 (0 forced) Flows exported: 0 in 0 packets (0 failures) Packets received by libpcap: 78049 Packets dropped by libpacp: 0 Packets dropped by interface: 0 # softflowctl expire-all expire-all softflowd[16769]: Expired 2770 flows. # softflowctl statistics statistics softflowd[16769]: Accumulated statistics: Number of active flows: 0 Packets processed: 78026 Fragments: 0 Ignored packets: 23 (23 non-IP, 0 too short) Flows expired: 2770 (0 forced) Flows exported: 2770 in 371 packets (0 failures) Packets received by libpcap: 78049 Packets dropped by libpacp: 0 Packets dropped by interface: 0 Expired flow statistics: minimum average maximum Flow bytes: 46 15935 2096331 Flow packets: 1 28 2802 Duration: 0.00s 4.01s 32.79s Expired flow reasons: tcp = 0 tcp.rst = 0 tcp.fin = 0 udp = 0 icmp = 0 general = 0 maxlife = 0 over 2Gb = 0 maxflows = 0 flushed = 2770 Per-protocol statistics: Octets Packets Avg Life Max Life Unknown (1): 15674 79 8.36s 31.82s Unknown (6): 40366416 67964 3.88s 32.76s Unknown (17): 3756178 9970 4.65s 32.79s Unknown (41): 884 13 4.72s 6.00s ## logged 852 lines There are still an aweful lot of flows getting lost :\ Regards, Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050928/9950aeed/attachment.bin From rbreathe at brookes.ac.uk Wed Sep 28 19:59:54 2005 From: rbreathe at brookes.ac.uk (Robin Breathe) Date: Wed, 28 Sep 2005 10:59:54 +0100 Subject: [netflow-tools] softflowd statistics bug In-Reply-To: References: <43392CD0.1010903@brookes.ac.uk> Message-ID: <433A699A.8050302@brookes.ac.uk> Damien Miller wrote: > On Tue, 27 Sep 2005, Robin Breathe wrote: >> In softflowd.c, it is falsely assumed that every expired flow will lead >> to 2 exports. >> This has made testing of the other bug (in flowd) a little harder as >> softflowd reports that it has exported more flows than it really has. >> >> Line 782: ft->flows_exported += num_expired * 2; >> Line 786: ft->flows_dropped += num_expired * 2; >> >> In the case of a unidirectional flow, this is not the case. > > I think this is fixed in the development version, of which I have just > set up public snapshot releases. Have a look at: > > http://www2.mindrot.org/softflowd_snap/ This provides better statistics, but still not what I'd expect. Softflowd exports two records per bi-directional flow, but only one record for a unidirectional flow. What `softflowctl statistics` displays as an "active flow" will actually equate to either one or two flow records being exported to the netflow collector depending upon whether it is uni- or bi-directional (e.g. blackholed probe versus normal tcp conversation). The old statistics assumed that every flow was bidirectional ("Flows exported" --> 2 * "Flows expired"), while the new ones record one expiry per flow independent of whether it is bidirectional or not ("Flows exported" --> 1 * "Flows expired"). Old: Number of active flows: 0 Packets processed: 101136 Fragments: 0 Ignored packets: 21 (21 non-IP, 0 too short) Flows expired: 1570 (0 forced) Flows exported: 3140 in 98 packets (0 failures) New: Number of active flows: 0 Packets processed: 78026 Fragments: 0 Ignored packets: 23 (23 non-IP, 0 too short) Flows expired: 2770 (0 forced) Flows exported: 2770 in 371 packets (0 failures) Packets received by libpcap: 78049 Packets dropped by libpacp: 0 Packets dropped by interface: 0 What I was suggesting was that it would be extremely useful to actually record the number of uni-directional flows (i.e. the items recorded by the collector) in the statistics. Something like the following: Flows exported: 2770 in 371 packets (0 failures, 5531 records) Regards, Robin -- Robin Breathe, Computer Services, Oxford Brookes University, Oxford, UK rbreathe at brookes.ac.uk Tel: +44 1865 483685 Fax: +44 1865 483073 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 187 bytes Desc: OpenPGP digital signature Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20050928/0fa18509/attachment.bin