From djm at mindrot.org Sun Jun 3 19:07:15 2007 From: djm at mindrot.org (Damien Miller) Date: Sun, 3 Jun 2007 19:07:15 +1000 (EST) Subject: [netflow-tools] softflowd and -m ? In-Reply-To: <465D91E6.2000700@pason.com> References: <465D91E6.2000700@pason.com> Message-ID: On Wed, 30 May 2007, Michael Gale wrote: > Hello, > > I am new to using netflows and am experimenting by using softflowd to > send tcpdump created files to ntop. > > Everything seems to be working except that I noticed a strange change > in stats when using the -m option in softflowd. > > With out specifying the -m, ntop reports 1.4GB of traffic with an > average of 136Mbps. If I run softflowd with "-m 1000000" ntop now > reports that 250MB of traffic was seen ? > > Does any one know why this would happen ? Two possibilites: 1. By cranking the number of flows tracked so high, you might be using up your RAM and pushing softflowd into swap. If softflowd swaps, then it will drop traffic. 2. By cranking up the number of trackable flows, you are giving long-lived flows more of a chance to stay in the list of tracked flows rather than being evicted by newer flows. Because ntop only receives notification of traffic once flows are evicted (and thus exported), it reports a lower traffic rate. If this is the case, you might want to set a "maxlife" timeout to force flows to be evicted every five minutes or so. -d From zhangjinxue at cernet.edu.cn Wed Jun 6 22:39:56 2007 From: zhangjinxue at cernet.edu.cn (=?gb2312?B?1cK9+tGn?=) Date: Wed, 6 Jun 2007 20:39:56 +0800 Subject: [netflow-tools] variable delay for log Message-ID: Hello, I am new to use flowd-0.9 to receive NetFlow v9 from Cisco router. Everything seems to be working except that I noticed when the flowd process is activated; there is a variable delay when the log file receives data. The delay is less than half an hour but usually nearly half an hour. My configuration is very simple: logfile "/mnt/sda5/flowd" listen on 0.0.0.0:2010 flow source 202.112.60.1 store ALL The environment is RedHat 9.0. Does any one know why does this variable delay occur? Thanks very much! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20070606/b3e1fb1b/attachment.html From djm at mindrot.org Fri Jun 15 09:13:10 2007 From: djm at mindrot.org (Damien Miller) Date: Fri, 15 Jun 2007 09:13:10 +1000 (EST) Subject: [netflow-tools] variable delay for log In-Reply-To: References: Message-ID: On Wed, 6 Jun 2007, ?????? wrote: > I am new to use flowd-0.9 to receive NetFlow v9 from Cisco router. > Everything seems to be working except that I noticed when the flowd process > is activated; there is a variable delay when the log file receives data. The > delay is less than half an hour but usually nearly half an hour. The delay occurs because your router will only send flows when they expire from its cache, which can be infrequent especially if your router is not busy. You can tweak this on your router by issuing commands like the following: ip flow-cache timeout active 5 This will cause your router to expire flows every five minutes. From memic at paniert.org Wed Jun 20 07:02:41 2007 From: memic at paniert.org (memic) Date: Tue, 19 Jun 2007 23:02:41 +0200 Subject: [netflow-tools] softflowd & openbsd carp devices In-Reply-To: <46481D05.9040904@paniert.org> References: <46307FDD.3020505@paniert.org> <46481D05.9040904@paniert.org> Message-ID: <46784471.30106@paniert.org> Hi, can the flows be send to one instance of flowd? memic wrote: > even with about 10 instances running? > > Damien Miller wrote: > >> On Thu, 26 Apr 2007, memic wrote: >> >> >> >>> Hi, >>> >>> i have an interface with softflowd running, but since this router is >>> going into a redudant setup >>> i will have carp devices. i will have serveral carp devices on one the >>> interfaces where i running >>> softflowd at the moment, because on this interfaces i have more than one ip. >>> does it make sence to run softflowd on the carp devices (serveral >>> softlfowd istances then) >>> or better to run it on the psyical interface like now (fxp0)? >>> >>> >> I would guess that it would be better to run it on the carp interfaces, >> but in a switched environment there should not be too much effective >> difference - the physical interface should not be seeing traffic destined >> for the virtual arp address of a carp interface when it is in slave mode. >> >> On the other hand, running multiple instances of softflowd shouldn't >> waste too many cycles. >> >> -d >> >> > > _______________________________________________ > netflow-tools mailing list > netflow-tools at mindrot.org > http://lists.mindrot.org/mailman/listinfo/netflow-tools > From srivi at endace.com Wed Jun 20 13:26:47 2007 From: srivi at endace.com (Srivi Ramanan) Date: Wed, 20 Jun 2007 15:26:47 +1200 Subject: [netflow-tools] Netflow headers Message-ID: <46789E77.5050107@endace.com> Does flowd stores the header while capturing to flows to a file ? From djm at mindrot.org Wed Jun 20 14:39:42 2007 From: djm at mindrot.org (Damien Miller) Date: Wed, 20 Jun 2007 14:39:42 +1000 (EST) Subject: [netflow-tools] softflowd & openbsd carp devices In-Reply-To: <46784471.30106@paniert.org> References: <46307FDD.3020505@paniert.org> <46481D05.9040904@paniert.org> <46784471.30106@paniert.org> Message-ID: On Tue, 19 Jun 2007, memic wrote: > Hi, > > can the flows be send to one instance of flowd? Sure - it can receive from multiple senders. -d From djm at mindrot.org Wed Jun 20 14:41:10 2007 From: djm at mindrot.org (Damien Miller) Date: Wed, 20 Jun 2007 14:41:10 +1000 (EST) Subject: [netflow-tools] Netflow headers In-Reply-To: <46789E77.5050107@endace.com> References: <46789E77.5050107@endace.com> Message-ID: On Wed, 20 Jun 2007, Srivi Ramanan wrote: > Does flowd stores the header while capturing to flows to a file ? Do you mean the netflow header? If so, some fields are preserved and are available in the log. They are the ones controlled by AGENT_INFO and FLOW_ENGINE_INFO in flowd.conf. -d From mizzou0 at yahoo.com Wed Jun 27 08:02:43 2007 From: mizzou0 at yahoo.com (David) Date: Tue, 26 Jun 2007 15:02:43 -0700 (PDT) Subject: [netflow-tools] Empty flowd log file Message-ID: <27495.15488.qm@web55501.mail.re4.yahoo.com> This is my first time to use flowd. After flowd installed and configured flow records can't write to the flowd log. File is always empty. flowd.conf content is here. # grep -v ^# /usr/local/etc/flowd.conf logfile "/var/log/flowd.bin" listen on 0.0.0.0:5004 flow source store ALL discard all I ran tcpdump and saw flow traffic is coming in from UDP port 5004. However the log file is still empty. I even modified the log file access to make it global writable. # ls -al /var/log/flow* -rw-rw-rw- 1 root root 0 Jun 26 13:08 /var/log/flowd.bin Can you please give me a hand to troubleshoot this issue? Thanks in advance, David. ____________________________________________________________________________________ TV dinner still cooling? Check out "Tonight's Picks" on Yahoo! TV. http://tv.yahoo.com/ From weweiss at mindspring.com Thu Jun 28 08:57:50 2007 From: weweiss at mindspring.com (Walter Weiss) Date: Wed, 27 Jun 2007 18:57:50 -0400 Subject: [netflow-tools] Empty log Message-ID: <001201c7b90e$96c14e00$a20698cd@your55e5f9e3d2> Hi; I have installed netflowd on the latest version of Fedora. It all seems to install ok. I have the following information from the command lines etc. But nothing ever writes to the log. Is there anything I can do to troubleshoot where the data is lost? Thanks Log entries in the messages file- Jun 27 13:17:26 flow_collector flowd[3403]: Received max number of packets (512) on fd 3 Jun 27 13:47:26 flow_collector flowd[3403]: Received max number of packets (512) on fd 3 Jun 27 14:17:26 flow_collector flowd[3403]: Received max number of packets (512) on fd 3 Jun 27 14:47:26 flow_collector flowd[3403]: Received max number of packets (512) on fd 3 Jun 27 15:17:26 flow_collector flowd[3403]: Received max number of packets (512) on fd 3 verified process is running [root at flow_collector etc]# ps -ef | grep flow avahi 2585 1 0 Jun26 ? 00:00:00 avahi-daemon: running [flowcollector.local] root 3402 1 0 Jun26 ? 00:00:00 flowd: monitor _flowd 3403 3402 0 Jun26 ? 00:00:00 flowd: net root 6065 6021 0 11:36 pts/0 00:00:00 grep flow [root at flow_collector etc]# ps -ef | grep syslog root 3370 1 0 Jun26 ? 00:00:00 syslogd -m 0 -a /var/empty/dev/log root 6067 6021 0 11:36 pts/0 00:00:00 grep syslog verified port seems to be listenting [root at flow_collector etc]# netstat -apn | grep flow udp 0 0 205.152.6.88:9995 0.0.0.0:* 3403/flowd: net unix 2 [ ] DGRAM 11493 3403/flowd: net unix 3 [ ] STREAM CONNECTED 11444 3403/flowd: net unix 3 [ ] STREAM CONNECTED 11443 3402/flowd: monitor checked for listening files and they seem to be ok [root at flow_collector etc]# lsof -i COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME portmap 2076 rpc 3u IPv4 6641 UDP *:sunrpc portmap 2076 rpc 4u IPv4 6642 TCP *:sunrpc (LISTEN) rpc.statd 2102 rpcuser 3w IPv4 6716 UDP *:1009 rpc.statd 2102 rpcuser 6u IPv4 6693 UDP *:1006 rpc.statd 2102 rpcuser 7u IPv4 6753 TCP *:1012 (LISTEN) hpiod 2376 root 0u IPv4 7305 TCP flow_collector:2208 (LISTEN) python 2381 root 4u IPv4 7321 TCP flow_collector:2207 (LISTEN) cupsd 2398 root 3u IPv4 7369 TCP flow_collector:ipp (LISTEN) cupsd 2398 root 5u IPv4 7372 UDP *:ipp sshd 2417 root 3u IPv6 7420 TCP *:ssh (LISTEN) sendmail 2442 root 4u IPv4 7513 TCP flow_collector:smtp (LISTEN) yum-updat 2568 root 8u IPv4 10063 TCP flow_collector:45266->admin.fedora.redhat.com:http (CLOSE_WAIT) avahi-dae 2585 avahi 13u IPv4 7822 UDP *:mdns avahi-dae 2585 avahi 14u IPv6 7823 UDP *:mdns avahi-dae 2585 avahi 15u IPv4 7824 UDP *:filenet-tms avahi-dae 2585 avahi 16u IPv6 7825 UDP *:filenet-rpc flowd 3403 _flowd 3u IPv4 11442 UDP flow_collector:palace-4 sshd 6019 root 3r IPv6 72128 TCP flow_collector:ssh->205.152.6.166:catchpole (ESTABLISHED) [root at flow_collector etc]# Seems to indicate I am getting packets [root at flow_collector /]# netflow v.9 packet (len 44) 1 recs, source 0x00000100 netflow v.9 options flowset output_flow_flush: flushing output queue len 0 Files associated with the flowd process [root at flow_collector /]# lsof -c flowd COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME flowd 6371 root cwd DIR 253,0 4096 7456541 /usr/local/sbin flowd 6371 root rtd DIR 253,0 4096 2 / flowd 6371 root txt REG 253,0 97616 7481639 /usr/local/sbin/flowd flowd 6371 root mem REG 253,0 46740 7816295 /lib/libnss_files-2.5.so flowd 6371 root mem REG 253,0 121684 7816881 /lib/ld-2.5.so flowd 6371 root mem REG 253,0 1576920 7816883 /lib/libc-2.5.so flowd 6371 root 0u CHR 1,3 2132 /dev/null flowd 6371 root 1u CHR 1,3 2132 /dev/null flowd 6371 root 2u CHR 136,1 3 /dev/pts/1 flowd 6371 root 4u unix 0xf731e800 18855 socket flowd 6373 _flowd cwd DIR 253,0 4096 425182 /var/empty flowd 6373 _flowd rtd DIR 253,0 4096 425182 /var/empty flowd 6373 _flowd txt REG 253,0 97616 7481639 /usr/local/sbin/flowd flowd 6373 _flowd mem REG 253,0 46740 7816295 /lib/libnss_files-2.5.so flowd 6373 _flowd mem REG 253,0 121684 7816881 /lib/ld-2.5.so flowd 6373 _flowd mem REG 253,0 1576920 7816883 /lib/libc-2.5.so flowd 6373 _flowd 0u CHR 1,3 2132 /dev/null flowd 6373 _flowd 1u CHR 1,3 2132 /dev/null flowd 6373 _flowd 2u CHR 136,1 3 /dev/pts/1 flowd 6373 _flowd 3u IPv4 18854 UDP flow_collector:palace-4 flowd 6373 _flowd 4u REG 253,0 0 230572 /usr/local/flowd/yort_test flowd 6373 _flowd 5u unix 0xf731e300 18856 socket flowd 6373 _flowd 6u unix 0xf731c300 18881 socket [root at flow_collector /]# I installed the debug info rpm but am not sure how to do anything with it. Here is the debug for the start up of the file. [root at flow_collector sbin]# flowd -d read_config: entering child_get_config: entering drop_privs: dropping privs without chroot send_config: entering fd = 4 send_config: done child_get_config: child config done recv_config: entering fd = 3 recv_config: ready to receive config Listener for [205.152.6.88]:9995 fd = 3 Increased socket receive buffer from 110592 to 524288 Setting socket send buf to 1024 privsep_init: entering drop_privs: dropping privs with chroot init_pfd: entering (num_fds = 0) init_pfd: done (num_fds = 2) client_open_log: entering answer_open_log: entering So the bottom line is I seem to be listening and receiving packets but nothing goes to the log. What can I do to troubleshoot further. Thanks Walt Weiss -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20070627/79b80c42/attachment-0001.html From joe.fortier at simondelivers.com Sat Jun 30 01:22:06 2007 From: joe.fortier at simondelivers.com (Josef Fortier) Date: Fri, 29 Jun 2007 10:22:06 -0500 Subject: [netflow-tools] Thanks and request for flowd Message-ID: <20070629152203.GA25011@simondelivers.com> First, thanks for a solid and simple flow collector. I've been running it for about 6 months, and it's been quite useful. I'm sure my usage is somewhat out of kilter with some of the intended functions. Here is what I've been doing 1) I collect everything. I'm not sure what I'll need to look at, so this seems the best policy. 2) I use flowd-reader to report. The perl interface indicates it's "just a thin wrapper" and I've not really looked at the Python interface. I've ended up with shell. The details a) I echo a filter list piped to flowd-reader with a -f flag to /dev/stdin. I wish there was a cleaner way to do ad-hoc filters. b) I pipe the output to awk to select fields, and then sort etc. to refine the output. QUESTIONS/REQUESTS 1) Is there a better way to pipe ad-hoc filters to flowd-reader (or another API). 2) Can tagging improve filtering. It appears that tagging is a way to create meta-information for reporting, but I keep wondering if I can use it to create positive additive filters ("find me all the http traffic, then find me the https") rather then negative filters (discard work fine cumulatively). Joe -- _______________________________________________________________________ Josef Fortier joe.fortier at simondelivers.com Network Administrator (763) 656-5650 _______________________________________________________________________