From djm at mindrot.org Tue Apr 8 15:19:20 2008 From: djm at mindrot.org (Damien Miller) Date: Tue, 8 Apr 2008 15:19:20 +1000 (EST) Subject: [netflow-tools] anoncvs.mindrot.org changes Message-ID: Hi, I have just converted anoncvs.mindrot.org from using GNU CVS to OpenCVS. The latter is quite a bit nicer to run as an anoncvs server, and it doesn't require write access to the repository like GNU CVS seems to. The only visible change from this should be that checkouts of particular tags or dates should now work (I didn't know they failed until today). If you experience any problems, please contact me. Thanks, Damien From JCourtney at unitedmcgill.com Fri Apr 18 00:19:13 2008 From: JCourtney at unitedmcgill.com (Joe Courtney) Date: Thu, 17 Apr 2008 10:19:13 -0400 Subject: [netflow-tools] softflowd & dd-wrt Message-ID: Does anyone have softflowd running successfully on an open-source dd-wrt firmware router? I am 99% done with getting it working, but I'm running into a problem "seeing" all the network traffic I wish to monitor. It is probably an IPTables or routing /filtering issue or how I've configured the IPs of my interfaces, but I can't seem to wrap my head around the exact problem. If I can figure this out, it is really going to be a nice way to send NetFlow data to a collector from a cheapo $50 router. I was able to install the Optware package for softflowd on the router (DD-WRT v24 RC-7 (03/13/08) std). I also installed tcpdump to make sure it wasn't an issue with softflowd (it isn't.) The problem is I'm only seeing broadcasts on the interface I'm monitoring (no traffic). (When setup on a full Linux box, I never had to do anything to the interface to monitor all the traffice.) For some reason on the dd-wrt, there is some kind of filtering happening that is preventing softflowd and tcpdump from seeing all the traffic. dd-wrt configuration: [WAN] --- Port 0 --------- Vlan1 ------- Eth0 [Port1] --- Port 1 -------- Vlan0 -------- Eth0 [Port2-4] - Port2-4 ------ Vlan0 * Default Vlan for non-tagged traffic So I have tried to use all the Ports for monitoring and many configurations of Vlans, but no luck. I have also tried putting the interfaces in permisc mode with ifconfig, but no luck. It seems the router is filtering all traffic. The firewall and gatway settings are off, the thing should be able to see all the traffic. ifconfig: br0s are for the Wireless (eth1. WLAN), that doesn't come into play for this config. Below I have added a second vlan, but no luck. br0 Link encap:Ethernet HWaddr 00:1A:70:FE:49:AE inet addr:192.168.0.11 Bcast:192.168.3.255 Mask:255.255.252.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:64919 errors:0 dropped:0 overruns:0 frame:0 TX packets:29719 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:4611647 (4.3 MiB) TX bytes:12926366 (12.3 MiB) br0:0 Link encap:Ethernet HWaddr 00:1A:70:FE:49:AE inet addr:169.254.255.1 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 eth0 Link encap:Ethernet HWaddr 00:1A:70:FE:49:AE UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:70842 errors:0 dropped:0 overruns:0 frame:0 TX packets:99384 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6515481 (6.2 MiB) TX bytes:18729587 (17.8 MiB) Interrupt:4 eth1 Link encap:Ethernet HWaddr 00:1A:70:FE:49:B0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:1051593 TX packets:56157 errors:2011 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:4376609 (4.1 MiB) Interrupt:2 Base address:0x5000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MULTICAST MTU:16436 Metric:1 RX packets:5896 errors:0 dropped:0 overruns:0 frame:0 TX packets:5896 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:566715 (553.4 KiB) TX bytes:566715 (553.4 KiB) vlan0 Link encap:Ethernet HWaddr 00:1A:70:FE:49:AE UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:60373 errors:0 dropped:0 overruns:0 frame:0 TX packets:29694 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:4454776 (4.2 MiB) TX bytes:7696374 (7.3 MiB) vlan1 Link encap:Ethernet HWaddr 00:1A:70:FE:49:AF inet addr:192.168.4.19 Bcast:192.168.4.23 Mask:255.255.255.248 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:392 errors:0 dropped:0 overruns:0 frame:0 TX packets:9269 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:24430 (23.8 KiB) TX bytes:428006 (417.9 KiB) vlan2 Link encap:Ethernet HWaddr 00:1A:70:FE:49:AE UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10087 errors:0 dropped:0 overruns:0 frame:0 TX packets:60451 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:765828 (747.8 KiB) TX bytes:9838153 (9.3 MiB) default IPTables: Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT 0 -- anywhere anywhere logdrop 0 -- anywhere anywhere state INVALID TCPMSS tcp -- anywhere anywhere tcp flags:SYN,RST/SYN tcpmss match 1461:65535 TCPMSS set 1460 lan2wan 0 -- anywhere anywhere ACCEPT 0 -- anywhere anywhere state RELATED,ESTABLISHED ACCEPT udp -- anywhere base-address.mcast.net/4 udp TRIGGER 0 -- anywhere anywhere TRIGGER type:in match:0 relate:0 trigger_out 0 -- anywhere anywhere ACCEPT 0 -- anywhere anywhere state NEW Chain logaccept (0 references) target prot opt source destination ACCEPT 0 -- anywhere anywhere Chain logdrop (1 references) target prot opt source destination DROP 0 -- anywhere anywhere Chain logreject (0 references) target prot opt source destination REJECT tcp -- anywhere anywhere tcp reject-with tcp-reset Thanks for any info. or comments. Joe Courtney ********************************************************** This email and any files transmitted with it are proprietary, confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. ********************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20080417/eb110618/attachment.html From djm at fuyu.mindrot.org Wed Apr 23 11:54:26 2008 From: djm at fuyu.mindrot.org (Damien Miller) Date: Wed, 23 Apr 2008 11:54:26 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: flowd Message-ID: <20080423015426.5E2C9A4F96@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: flowd Changes by: djm at fuyu.mindrot.org 08/04/23 11:54:26 Modified files: . : ChangeLog flowd-reader.c store.c store.h Log message: - (djm) Support flow-tools CSV output format in flowd-reader. Patch from weinhold AT berbee.com Diff commands: cvs -nQq rdiff -u -r1.170 -r1.171 flowd/ChangeLog cvs -nQq rdiff -u -r1.22 -r1.23 flowd/flowd-reader.c cvs -nQq rdiff -u -r1.38 -r1.39 flowd/store.c cvs -nQq rdiff -u -r1.30 -r1.31 flowd/store.h CVSWeb: http://cvsweb.mindrot.org/index.cgi/flowd/ChangeLog?r1=1.170;r2=1.171 http://cvsweb.mindrot.org/index.cgi/flowd/flowd-reader.c?r1=1.22;r2=1.23 http://cvsweb.mindrot.org/index.cgi/flowd/store.c?r1=1.38;r2=1.39 http://cvsweb.mindrot.org/index.cgi/flowd/store.h?r1=1.30;r2=1.31 Please note that there may be a delay before commits are available on the public CVSWeb site. From djm at fuyu.mindrot.org Wed Apr 23 12:01:04 2008 From: djm at fuyu.mindrot.org (Damien Miller) Date: Wed, 23 Apr 2008 12:01:04 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: flowd Message-ID: <20080423020104.A111BA4F96@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: flowd Changes by: djm at fuyu.mindrot.org 08/04/23 12:01:04 Modified files: . : privsep.c Log message: wrap Diff commands: cvs -nQq rdiff -u -r1.32 -r1.33 flowd/privsep.c CVSWeb: http://cvsweb.mindrot.org/index.cgi/flowd/privsep.c?r1=1.32;r2=1.33 Please note that there may be a delay before commits are available on the public CVSWeb site. From djm at mindrot.org Wed Apr 23 12:02:13 2008 From: djm at mindrot.org (Damien Miller) Date: Wed, 23 Apr 2008 12:02:13 +1000 (EST) Subject: [netflow-tools] flowd is terrific! What do the continuous syslog messages mean? In-Reply-To: <96CACD3D-918A-42FA-A23A-6F21E7C73391@briworks.com> References: <96CACD3D-918A-42FA-A23A-6F21E7C73391@briworks.com> Message-ID: Hi, Sorry for taking a little while to get back to you. Could you please try this patch? (The messages are harmless) Index: flowd.c =================================================================== RCS file: /var/cvs/flowd/flowd.c,v retrieving revision 1.77 diff -u -p -r1.77 flowd.c --- flowd.c 24 Oct 2007 01:04:10 -0000 1.77 +++ flowd.c 23 Apr 2008 02:00:52 -0000 @@ -1185,7 +1185,7 @@ receive_many(struct flowd_config *conf, for (i = 0; i < INPUT_MAX_PACKET_PER_FD; i++) { if (receive_packet(conf, peers, net_fd) == 0) { - syslog(LOG_DEBUG, "Received max number of packets " + logit(LOG_DEBUG, "Received max number of packets " "(%d) on fd %d", INPUT_MAX_PACKET_PER_FD, net_fd); return; } On Fri, 28 Mar 2008, Jeff Saxe wrote: > My apologies for first sending this directly to the author instead of to a > proper mailing list, which he took the time to set up. I am resending it to > the list. > > > Good day! I'm a network engineer, smart guy, and Perl hacker at a smallish ISP > in Charlottesville, Virginia, USA, and I'm trying to use flowd to put together > a clever little client billing system. I believe everything is working fine, > but when I turn on several routers' and several interfaces' worth of NetFlow > packets toward this (reasonably powerful) Linux box at the same time, I get > syslog messages like... > > > Mar 26 20:13:27 chance flowd[17102]: Received max number of packets (512) on > fd 3 > Mar 26 20:13:27 chance flowd[17102]: Valid netflow v.5 packet 30 flows > Mar 26 20:13:27 chance flowd[17102]: Received max number of packets (512) on > fd 3 > Mar 26 20:13:27 chance flowd[17102]: Valid netflow v.5 packet 30 flows > Mar 26 20:13:27 chance flowd[17102]: Valid netflow v.5 packet 30 flows > Mar 26 20:13:27 chance flowd[17102]: Valid netflow v.5 packet 30 flows > > Does this mean that actual NetFlow data are being discarded because they are > arriving too fast? I should warn you that I'm using a lot of flowd.conf > "accept" rules (on the order of 1,300, and more coming next week), so is that > a problem? I mainly want to know if all the data are being collected or not; > if they are, I will recompile the code to just suppress this warning message, > but if it's a real problem, I'd like to know if you have any suggestions for > not dropping packets. I've attached the current flowd.conf in case you are > interested, and if you want to see the Perl code or MySQL table structure > behind the automated rule generation, I can show you that, too. > > ? > > If you don't have time to help a stranger with this for free, I understand; > just please tell me if the code is throwing away flows or not. Thanks very > much, sir! > > -- Jeff Saxe, Network Engineer > Blue Ridge InternetWorks, Charlottesville, VA > CCIE # 9376 > 434-817-0707 ext. 2024 (work) / 434-882-3508 (cell) / JSaxe at briworks.com > > > > From djm at fuyu.mindrot.org Wed Apr 23 12:02:32 2008 From: djm at fuyu.mindrot.org (Damien Miller) Date: Wed, 23 Apr 2008 12:02:32 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: flowd Message-ID: <20080423020232.B09AAA4F98@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: flowd Changes by: djm at fuyu.mindrot.org 08/04/23 12:02:32 Modified files: . : ChangeLog flowd.c Log message: - (djm) Use proper API for logging debug information; Spotted by JSaxe AT briworks.com Diff commands: cvs -nQq rdiff -u -r1.171 -r1.172 flowd/ChangeLog cvs -nQq rdiff -u -r1.77 -r1.78 flowd/flowd.c CVSWeb: http://cvsweb.mindrot.org/index.cgi/flowd/ChangeLog?r1=1.171;r2=1.172 http://cvsweb.mindrot.org/index.cgi/flowd/flowd.c?r1=1.77;r2=1.78 Please note that there may be a delay before commits are available on the public CVSWeb site.