From fboehm at aon.at Mon Jan 5 10:14:58 2009 From: fboehm at aon.at (=?ISO-8859-15?Q?Franz_B=F6hm?=) Date: Mon, 05 Jan 2009 00:14:58 +0100 Subject: [netflow-tools] Weird duplicate netflow records Message-ID: <496142F2.90505@aon.at> Please have a look at the following netflow records. Sometimes I get double records like the samples below. They were generated with pfflowd, collected with nfcapd and viewed with nfdump. 2009-01-04 11:00:26.556 5167.000 TCP 10.0.3.34:4147 -> 80.140.195.57:30730 8118 9.4 M 1 2009-01-04 11:00:26.556 5167.000 TCP 80.140.195.57:30730 -> 10.0.3.34:4147 4583 188560 1 2009-01-04 11:00:25.990 5178.000 TCP 10.0.3.34:4147 -> 80.140.195.57:30730 8118 9.4 M 1 2009-01-04 11:00:25.990 5178.000 TCP 80.140.195.57:30730 -> 10.0.3.34:4147 4583 188560 1 2009-01-04 14:25:26.720 800.000 TCP 10.0.3.50:1942 -> 87.248.217.89:80 19858 802352 1 2009-01-04 14:25:26.720 800.000 TCP 87.248.217.89:80 -> 10.0.3.50:1942 38147 53.9 M 1 2009-01-04 14:25:25.720 801.000 TCP 10.0.3.50:1942 -> 87.248.217.89:80 19858 802352 1 2009-01-04 14:25:25.720 801.000 TCP 87.248.217.89:80 -> 10.0.3.50:1942 38147 53.9 M 1 I would be very thankful if someone has a hint for me. From cstamas at digitus.itk.ppke.hu Mon Jan 5 12:13:21 2009 From: cstamas at digitus.itk.ppke.hu (Csillag Tamas) Date: Mon, 5 Jan 2009 02:13:21 +0100 Subject: [netflow-tools] Weird duplicate netflow records In-Reply-To: <496142F2.90505@aon.at> References: <496142F2.90505@aon.at> Message-ID: <20090105011318.GX10810@rivendell> On Mon, Jan 05, 2009 at 12:14:58AM +0100, Franz B?hm wrote: > Please have a look at the following netflow records. Sometimes I get > double records like the samples below. > They were generated with pfflowd, collected with nfcapd and viewed with > nfdump. > > 2009-01-04 11:00:26.556 5167.000 TCP 10.0.3.34:4147 -> > 80.140.195.57:30730 8118 9.4 M 1 > 2009-01-04 11:00:26.556 5167.000 TCP 80.140.195.57:30730 -> > 10.0.3.34:4147 4583 188560 1 > 2009-01-04 11:00:25.990 5178.000 TCP 10.0.3.34:4147 -> > 80.140.195.57:30730 8118 9.4 M 1 > 2009-01-04 11:00:25.990 5178.000 TCP 80.140.195.57:30730 -> > 10.0.3.34:4147 4583 188560 1 > > 2009-01-04 14:25:26.720 800.000 TCP 10.0.3.50:1942 -> > 87.248.217.89:80 19858 802352 1 > 2009-01-04 14:25:26.720 800.000 TCP 87.248.217.89:80 -> > 10.0.3.50:1942 38147 53.9 M 1 > 2009-01-04 14:25:25.720 801.000 TCP 10.0.3.50:1942 -> > 87.248.217.89:80 19858 802352 1 > 2009-01-04 14:25:25.720 801.000 TCP 87.248.217.89:80 -> > 10.0.3.50:1942 38147 53.9 M 1 > > I would be very thankful if someone has a hint for me. Just guessing: Are the states bound to one interface or two interfaces? Regards, cstamas -- CSILLAG Tamas (cstamas) - http://digitus.itk.ppke.hu/~cstamas The present need for security products far exceeds the number of individuals capable of designing secure systems. Consequently, industry has resorted to employing folks and purchasing "solutions" from vendors that shouldn't be let near a project involving securing a system. -- Lucky Green From djm at mindrot.org Mon Jan 5 18:24:25 2009 From: djm at mindrot.org (Damien Miller) Date: Mon, 5 Jan 2009 18:24:25 +1100 (EST) Subject: [netflow-tools] softflowctl statistics In-Reply-To: <20081207175244.GA85769@bewilderbeast.blackhelicopters.org> References: <20081207175244.GA85769@bewilderbeast.blackhelicopters.org> Message-ID: On Sun, 7 Dec 2008, Michael W. Lucas wrote: > Hi, > > (Please pardon the nitpicking questions. My new book includes a > section on softflowd. While I've used softflowd happily for years, > this means that I'm now poking into corners that I've never looked at > before.) Excellent! > I'm taking a close look at "softflowctl statistics," and something > doesn't seem quite right to me. Here's output from 0.9.8 on a FreeBSD > 6.4 test box: > > softflowd[61946]: Accumulated statistics: Number of active flows: 16 > Packets processed: 11898 Fragments: 0 Ignored packets: 46 (46 non-IP, > 0 too short) Flows expired: 759 (0 forced) Flows exported: 784 in > 67 packets (0 failures) Packets received by libpcap: 12156 Packets > dropped by libpcap: 0 Packets dropped by interface: 3217012028 > > I would expect "packets processed" + "ignored packets" = "packets > received by libpcap". This rarely seems to be the case. Does softflowd > lose packets? Hopefully not :) The "Packets received|dropped by ..." are statistics that are provided by libpcap and the accounting there might be slightly different. In particular, it might be that there are a few packets sitting in a pcap buffer waiting to be sent to softflowd. It's also possible that I made a math error somewhere in softflowd. softflowd can lose packets when it is too busy to process incoming ones. It is possible to replicate this by running "softflowctl expire-all" in a tight loop. > Also, we've expired 759 flows, but exported 784. Where did the extra > 25 flows come from? This is some terminology confusion that is entirely my fault: the "flows" in the "expired" statistic are softflowd's internal flow representation, which are bidirectional. The "flows" in the "exported" statistic are NetFlow flows, which are unidirectional. If every flow had bidirectional traffic the latter should be 2 x the former, but this is not always the case (e.g. broadcast packets, packets dropped by firewalls). > What makes softflowd "force" expiration of a flow? "softflowctl > expire-all" doesn't seem to increment this counter. This is when flows are "forced out" before their timeout expires by new flows. Forcible expiry is performed on LRU (least recently used) flows. You should be able to demonstrate forcible expiry by setting an unreasonably low number of maximum flows (-m 16 or somesuch). > Last, why would an interface on a machine with very little traffic > show 3217012028 packets dropped by the interface? That value comes from libpcap, but it looks like it was either uninitialised or has underflowed an unsigned integer. -d PS. sorry for the slow reply - December was a mad month... From djm at mindrot.org Mon Jan 5 18:25:53 2009 From: djm at mindrot.org (Damien Miller) Date: Mon, 5 Jan 2009 18:25:53 +1100 (EST) Subject: [netflow-tools] netflow problem In-Reply-To: <53834cf20812231213k6910297bq73d874b68301bb39@mail.gmail.com> References: <53834cf20812231213k6910297bq73d874b68301bb39@mail.gmail.com> Message-ID: Hi, It looks like the directory that flowd is trying to create a pid file in does not exist. Try setting pidfile "/var/run/flowd.pid" in flowd.conf or somesuch. -d On Tue, 23 Dec 2008, Jaime Blasco wrote: > Hi, I have install flowd-0.9.1, > > I have some problems running flowd, when I run flowd with -d option this > is the output: > > read_config: entering > child_get_config: entering > drop_privs: dropping privs without chroot > send_config: entering fd = 4 > send_config: done > child_get_config: child config done > recv_config: entering fd = 3 > recv_config: ready to receive config > Listener for [127.0.0.1]:12345 fd = 3 > Adjusted socket receive buffer from 112640 to 524288 > Setting socket send buf to 1024 > Listener for [::1]:12345 fd = 4 > Adjusted socket receive buffer from 112640 to 524288 > Setting socket send buf to 1024 > privsep_init: entering > fopen(/usr/local/var/run/flowd.pid): No such file or directorydrop_privs: > dropping privs with chroot > > init_pfd: entering (num_fds = 0) > init_pfd: done (num_fds = 3) > client_open_log: entering > receive_fd: recvmsg: Connection reset by peer > > Anyone can help me? > > Regards > -- > _______________________________ > > Jaime Blasco > > www.ossim.com > www.alienvault.com > Email: jaime.blasco at alienvault.com > > > From djm at mindrot.org Mon Jan 5 18:29:20 2009 From: djm at mindrot.org (Damien Miller) Date: Mon, 5 Jan 2009 18:29:20 +1100 (EST) Subject: [netflow-tools] Weird duplicate netflow records In-Reply-To: <496142F2.90505@aon.at> References: <496142F2.90505@aon.at> Message-ID: On Mon, 5 Jan 2009, Franz B?hm wrote: > Please have a look at the following netflow records. Sometimes I get > double records like the samples below. > They were generated with pfflowd, collected with nfcapd and viewed with > nfdump. I'm not sure what could be causing this - pfflowd should only send duplicate-looking flows when it encounters expired pf states that have recorded more traffic that will fit in a 32-bit integer. Can you correlate the records with a tcpdump on the pfsync interface that pfflowd is listening to? That will tell you whether the duplicate flows are coming from pfflowd or pfsync. -d > 2009-01-04 11:00:26.556 5167.000 TCP 10.0.3.34:4147 -> > 80.140.195.57:30730 8118 9.4 M 1 > 2009-01-04 11:00:26.556 5167.000 TCP 80.140.195.57:30730 -> > 10.0.3.34:4147 4583 188560 1 > 2009-01-04 11:00:25.990 5178.000 TCP 10.0.3.34:4147 -> > 80.140.195.57:30730 8118 9.4 M 1 > 2009-01-04 11:00:25.990 5178.000 TCP 80.140.195.57:30730 -> > 10.0.3.34:4147 4583 188560 1 > > 2009-01-04 14:25:26.720 800.000 TCP 10.0.3.50:1942 -> > 87.248.217.89:80 19858 802352 1 > 2009-01-04 14:25:26.720 800.000 TCP 87.248.217.89:80 -> > 10.0.3.50:1942 38147 53.9 M 1 > 2009-01-04 14:25:25.720 801.000 TCP 10.0.3.50:1942 -> > 87.248.217.89:80 19858 802352 1 > 2009-01-04 14:25:25.720 801.000 TCP 87.248.217.89:80 -> > 10.0.3.50:1942 38147 53.9 M 1 > > I would be very thankful if someone has a hint for me. > _______________________________________________ > netflow-tools mailing list > netflow-tools at mindrot.org > https://lists.mindrot.org/mailman/listinfo/netflow-tools > From eugene at edm.ie Fri Jan 9 08:32:52 2009 From: eugene at edm.ie (Eugene McNamee) Date: Thu, 8 Jan 2009 21:32:52 -0000 Subject: [netflow-tools] softflowd+WD MyBook Message-ID: <002401c971d8$a9988730$fcc99590$@ie> Hi, I have installed softflowd on a Western Digital MyBook World Edition: Processor : ARM926EJ-Sid(wb) rev 5 (v5l) RAM : Hynix HY50U561622E (30032 kB) Kernel : BusyBox 2.6.17.14 Webserver : lighttpd Everything is installed correctly. Unfortunately I haven't been able to find any documentation that can help me with softflowd. I wish to be able to have softflowd capture netflows from my LinkSys 160N router. Currently I am using ManageEngine's NetFlow Analyser on a Vista box but I need an always on 24/7 reporting facility and as MyBook is directly cabled to router it seems to me the simplest option. The router is running dd-wrt v24-sp1 std-special and is exporting snmp data on port 9996. Lighttpd is running and there is a public folder available for data. Can anyone offer any suggestions as to how to configure softflowd to capture data and export it and then how to read this data from a windows box via http??? Many Thanks, Eugene