From subra.tech at gmail.com Tue Jul 10 05:20:50 2007 From: subra.tech at gmail.com (subramanian ramasamy) Date: Mon, 9 Jul 2007 12:20:50 -0700 Subject: [netflow-tools] Input for softflowd from pcap file. Message-ID: <77c4bc660707091220x95cc920r35b1e71a95b2a868@mail.gmail.com> Hi, Iam new to NetFlow and softflowd. I ran softflowd with input from a pcap file which has a 20 min complete ssh conversation between two machines. I ran tcpdump on the collector machine and saved the NetFlow V9 traffic from softflowd. I saved the exported info as dmp file and later examined using wireshark. I only see Template Flowset and no Data Flowset. Am i doing anything wrong ? Thanks, Subra. > softflowd -D -v 9 -r TCP_20min_conn.dmp -n 10.6.100.134:9992 softflowd v0.9.8 starting data collection Exporting flows to [10.6.100.134]:9992 ADD FLOW seq:1 [10.1.1.40]:22 <> [10.1.5.46]:3123 proto:6 Shutting down after pcap EOF Shutting down on user request Starting expiry scan: mode -1 Queuing flow seq:1 (0x927d4c8) for expiry reason 3 Finished scan 1 flow(s) to be evicted Flow 2/0: r 0 offset 190 type 0004 len 66(0x0042) flows 2 Sending flow packet len = 192 sent 1 netflow packets EXPIRED: seq:1 [10.1.1.40]:22 <> [10.1.5.46]:3123 proto:6 octets>:5143 packets>:48 octets<:6324 packets<:46 start:2007-04-30T22:18:59.801finish:2007-04-30T22:43: 13.317 tcp>:1b tcp<:1b flowlabel>:00000000 flowlabel<:00000000 (0x927d4c8) Number of active flows: 0 Packets processed: 94 Fragments: 0 Ignored packets: 0 (0 non-IP, 0 too short) Flows expired: 1 (0 forced) Flows exported: 1 in 1 packets (0 failures) Expired flow statistics: minimum average maximum Flow bytes: 11467 11467 11467 Flow packets: 94 94 94 Duration: 1453.52s 1453.52s 1453.52s Expired flow reasons: tcp = 0 tcp.rst = 0 tcp.fin = 0 udp = 0 icmp = 0 general = 0 maxlife = 0 over 2Gb = 0 maxflows = 0 flushed = 1 Per-protocol statistics: Octets Packets Avg Life Max Life tcp (6): 11467 94 1453.52s 1453.52s -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20070709/db2c3e54/attachment.html From zhangjinxue at cernet.edu.cn Tue Jul 10 12:29:33 2007 From: zhangjinxue at cernet.edu.cn (=?gb2312?B?1cK9+tGn?=) Date: Tue, 10 Jul 2007 10:29:33 +0800 Subject: [netflow-tools] A small bug Message-ID: Dear Mr. Miller, When I ran flowd-0.9 in my Sparc64 box, I found that when a v9 template arrives, the same DEBUG info will show: ?short netflow v.9 flowset template 0x00000011/0x00000104 88 bytes from *.*.*.*?, so I can?t receive data flowsets It is abnormal because I can receive netflow v9 data in my 32-bits box by flowd-0.9. Through debugging, I think the 883th line in flowd.c maybe incorrect since ?tmplr? is a pointer or address. So sizeof(tmplr)= 4 in 32-bits box; =8 in 64-bits box; When I rewrite it as sizeof(*tmplr), it work normally both in 32-bits and 64-bits box. 879 tmplr = (struct NF9_TEMPLATE_FLOWSET_RECORD *) 880 (pkt + offset); 881 recs[i].type = ntohs(tmplr->type); 882 recs[i].len = ntohs(tmplr->length); 883 offset += sizeof(tmplr); 884 #ifdef DEBUG_NF9 885 logit(LOG_DEBUG, " record %d: type %d len %d", 886 i, recs[i].type, recs[i].len); 887 #endif Best wishes Jinxue Zhang -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.mindrot.org/pipermail/netflow-tools/attachments/20070710/20b65e82/attachment.html From djm at mindrot.org Mon Jul 23 16:24:55 2007 From: djm at mindrot.org (Damien Miller) Date: Mon, 23 Jul 2007 16:24:55 +1000 (EST) Subject: [netflow-tools] Empty log In-Reply-To: <001201c7b90e$96c14e00$a20698cd@your55e5f9e3d2> References: <001201c7b90e$96c14e00$a20698cd@your55e5f9e3d2> Message-ID: On Wed, 27 Jun 2007, Walter Weiss wrote: > Hi; > > I have installed netflowd on the latest version of Fedora. It > all seems to install ok. I have the following information from the command > lines etc. But nothing ever writes to the log. Is there anything I can do > to troubleshoot where the data is lost? Thanks > > [root at flow_collector /]# netflow v.9 packet (len 44) 1 recs, source > 0x00000100 > netflow v.9 options flowset > output_flow_flush: flushing output queue len 0 Flowd doesn't support NetFlow v.9 "options flowsets". When I wrote the Netflow v.9 code, I didn't have access to anything that generated them and I recall finding the documentation on them somewhat unclear. If these are all you are receiving, then you won't see anything logged. Otherwise, could you send me your flowd.conf? -d From djm at mindrot.org Mon Jul 23 16:28:15 2007 From: djm at mindrot.org (Damien Miller) Date: Mon, 23 Jul 2007 16:28:15 +1000 (EST) Subject: [netflow-tools] Thanks and request for flowd In-Reply-To: <20070629152203.GA25011@simondelivers.com> References: <20070629152203.GA25011@simondelivers.com> Message-ID: On Fri, 29 Jun 2007, Josef Fortier wrote: > QUESTIONS/REQUESTS > > 1) Is there a better way to pipe ad-hoc filters to flowd-reader (or > another API). What sort of filters are you after? I wouldn't oppose allowing some basic commandline switches to filter by source/destination address or port. > 2) Can tagging improve filtering. It appears that tagging is a way > to create meta-information for reporting, but I keep wondering if > I can use it to create positive additive filters ("find me all > the http traffic, then find me the https") rather then negative > filters (discard work fine cumulatively). Yes, there have been other requesting that too. I haven't been able to figure out a syntax for flowd.conf that works well for additive/cumulative tagging. The problem is that the filters are now "one match wins", but cumulative tagging breaks that a little. Suggestions welcome! -d From djm at mindrot.org Mon Jul 23 16:30:20 2007 From: djm at mindrot.org (Damien Miller) Date: Mon, 23 Jul 2007 16:30:20 +1000 (EST) Subject: [netflow-tools] Input for softflowd from pcap file. In-Reply-To: <77c4bc660707091220x95cc920r35b1e71a95b2a868@mail.gmail.com> References: <77c4bc660707091220x95cc920r35b1e71a95b2a868@mail.gmail.com> Message-ID: On Mon, 9 Jul 2007, subramanian ramasamy wrote: > Hi, > > Iam new to NetFlow and softflowd. > > I ran softflowd with input from a pcap file which has a 20 min complete ssh > conversation between two machines. I ran tcpdump on the collector machine > and saved the NetFlow V9 traffic from softflowd. I saved the exported info > as dmp file and later examined using wireshark. I only see Template Flowset > and no Data Flowset. > > Am i doing anything wrong ? I have no idea - this is a pretty convoluted way to look at flows. Softflowd is definitely seeing the flow, and appears to be exporting it. Could you set up some NetFlow (e.g. flowd) and try to capture it? -d From djm at fuyu.mindrot.org Mon Jul 23 16:33:42 2007 From: djm at fuyu.mindrot.org (Damien Miller) Date: Mon, 23 Jul 2007 16:33:42 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: flowd Message-ID: <20070723063342.58F2E3C6B2@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: flowd Changes by: djm at fuyu.mindrot.org 07/07/23 16:33:42 Modified files: . : ChangeLog flowd.c Log message: - (djm) Fix NetFlow v.9 flowset template parsing on LP!32 platforms. Report and fix from zhangjinxue AT cernet.edu.cn Diff commands: cvs -nQq rdiff -u -r1.166 -r1.167 flowd/ChangeLog cvs -nQq rdiff -u -r1.74 -r1.75 flowd/flowd.c CVSWeb: http://cvsweb.mindrot.org/index.cgi/flowd/ChangeLog?r1=1.166;r2=1.167 http://cvsweb.mindrot.org/index.cgi/flowd/flowd.c?r1=1.74;r2=1.75 Please note that there may be a delay before commits are available on the public CVSWeb site. From djm at mindrot.org Mon Jul 23 16:37:01 2007 From: djm at mindrot.org (Damien Miller) Date: Mon, 23 Jul 2007 16:37:01 +1000 (EST) Subject: [netflow-tools] A small bug In-Reply-To: References: Message-ID: On Tue, 10 Jul 2007, zhangjinxue ? wrote: > Dear Mr. Miller, > > When I ran flowd-0.9 in my Sparc64 box, I found that when a v9 > template arrives, the same DEBUG info will show: > > ??short netflow v.9 flowset template > 0x00000011/0x00000104 88 bytes from *.*.*.*??, so I can??t receive data > flowsets > > It is abnormal because I can receive netflow v9 data in my 32-bits > box by flowd-0.9. Through debugging, I think the 883th line in flowd.c > maybe incorrect since ??tmplr?? is a pointer or address. > > So > > sizeof(tmplr)= 4 in 32-bits box; > > =8 in 64-bits box; > > > > When I rewrite it as sizeof(*tmplr), it work normally both in 32-bits and > 64-bits box. Great work! Your fix is correct and I have just committed it. It will be in the next release of flowd. Thanks, Damien Miller From joe.fortier at simondelivers.com Tue Jul 24 02:56:38 2007 From: joe.fortier at simondelivers.com (Josef Fortier) Date: Mon, 23 Jul 2007 11:56:38 -0500 Subject: [netflow-tools] Thanks and request for flowd In-Reply-To: References: <20070629152203.GA25011@simondelivers.com> Message-ID: <20070723165635.GA3812@simondelivers.com> Hi Damien: Thanks for the reply. > > 1) Is there a better way to pipe ad-hoc filters to flowd-reader (or > > another API). > > What sort of filters are you after? I wouldn't oppose allowing some basic > commandline switches to filter by source/destination address or port. First, I should clarify my assumptions.... I'm rotating flowd files once a week. When I run the tools/ scripts (wormsuspects.pl etc) they take a while. My assumption is that the time reflects a linear search on the flowd file. Is this correct? The flowd-reader lets me specify a config file, which in turn can specify filters. This seems to have much better performance. Looking at Flowd.pm it looks like there is no facility for filtering. In practice, this appears to give me a large speed boost. I'd suspect this because flowd-reader can throw away things much faster then an external program... Right now, I'm using short shell script to drive flowd-reader, and then pump the output into something else (perl, awk) for further processing. Example: echo " # Comment regarding filters discard quick before date .......... discard quick src ! [local-address-range] " | flowd-reader /dev/stdin \ | perl -e ' # a program to do further processing ' This works OK, but the syntax is a little gunky (in particular the /dev/stdin bothers me). One options is to attach stdin and read filters from there (a flag maybe). For the speed reasons, I'd like to be able to pass ad-hoc filter statement lists into whatever reads the flowd file. My wish is to get fast, relatively straightforward scripted reports out. It's all about reporting... > > > 2) Can tagging improve filtering. It appears that tagging is a way > > to create meta-information for reporting, but I keep wondering if > > I can use it to create positive additive filters ("find me all > > the http traffic, then find me the https") rather then negative > > filters (discard work fine cumulatively). > > Yes, there have been other requesting that too. I haven't been able to > figure out a syntax for flowd.conf that works well for additive/cumulative > tagging. The problem is that the filters are now "one match wins", but > cumulative tagging breaks that a little. Suggestions welcome! I've worked out how to get some of this from the filter statements. Basically, 1) Filter out all the negative stuff 2) use "accept quick" to add to the positive filter 3) discard everything Example discard quick before date [date_number] accept quick src [address_range_ONE] accept quick src [address_range_TWO] discard any This is limited in that the accept statements still need to be atomic, no cumulative statements. Thanks Joe -- _______________________________________________________________________ Josef Fortier Network Administrator _______________________________________________________________________ From fweimer at bfk.de Wed Jul 25 00:40:34 2007 From: fweimer at bfk.de (Florian Weimer) Date: Tue, 24 Jul 2007 16:40:34 +0200 Subject: [netflow-tools] Some softflowd patches Message-ID: <82d4yhj259.fsf@mid.bfk.de> Here are a few patches for softflowd. Without LOG_NDELAY in the openlog call, the actual open will be delayed until the first syslog call. At that point, it might fail because of the previous chroot call. The "volatile" hunk is probably not required because the compiler needs to assume that the function pointer escapes from the signal function anyway (it should be "volatile sig_atomic_t", but I don't know if this is sufficiently portable). The for(;;) patch is required for proper SIGTERM support (otherwise, softflowd never reacts to SIGTERM, at least on Linux). --- ../tmp/log.c 2004-09-10 11:08:08.000000000 +0200 +++ log.c 2007-07-24 14:18:23.000000000 +0200 @@ -38,7 +38,7 @@ if (to_stderr) logstderr = 1; else - openlog(PROGNAME, LOG_PID, LOG_DAEMON); + openlog(PROGNAME, LOG_PID | LOG_NDELAY, LOG_DAEMON); } void --- ../tmp/softflowd.c 2006-11-02 07:23:29.000000000 +0100 +++ softflowd.c 2007-07-24 15:18:30.000000000 +0200 @@ -57,7 +57,7 @@ static int verbose_flag = 0; /* Debugging flag */ /* Signal handler flags */ -static int graceful_shutdown_request = 0; +static volatile int graceful_shutdown_request = 0; /* Context for libpcap callback functions */ struct CB_CTXT { @@ -1824,7 +1824,7 @@ cb_ctxt.ft = &flowtrack; cb_ctxt.linktype = linktype; cb_ctxt.want_v6 = target.dialect->v6_capable || always_v6; - for(;;) { + while (!graceful_shutdown_request) { /* * Silly libpcap's timeout function doesn't work, so we * do it here (only if we are reading live) -- Florian Weimer BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstra?e 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 From djm at fuyu.mindrot.org Wed Jul 25 09:48:58 2007 From: djm at fuyu.mindrot.org (Damien Miller) Date: Wed, 25 Jul 2007 09:48:58 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: softflowd Message-ID: <20070724234858.3BFF03C6B2@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: softflowd Changes by: djm at fuyu.mindrot.org 07/07/25 09:48:58 Modified files: . : ChangeLog softflowd.c Log message: - (djm) KNF Diff commands: cvs -nQq rdiff -u -r1.89 -r1.90 softflowd/ChangeLog cvs -nQq rdiff -u -r1.93 -r1.94 softflowd/softflowd.c CVSWeb: http://cvsweb.mindrot.org/index.cgi/softflowd/ChangeLog?r1=1.89;r2=1.90 http://cvsweb.mindrot.org/index.cgi/softflowd/softflowd.c?r1=1.93;r2=1.94 Please note that there may be a delay before commits are available on the public CVSWeb site. From djm at fuyu.mindrot.org Wed Jul 25 09:49:43 2007 From: djm at fuyu.mindrot.org (Damien Miller) Date: Wed, 25 Jul 2007 09:49:43 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: softflowd Message-ID: <20070724234943.E4A5B3C6B2@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: softflowd Changes by: djm at fuyu.mindrot.org 07/07/25 09:49:43 Modified files: . : ChangeLog softflowd.c Log message: - (djm) Correctly exit from mainloop on signal - patch from Florian Weimer Diff commands: cvs -nQq rdiff -u -r1.90 -r1.91 softflowd/ChangeLog cvs -nQq rdiff -u -r1.94 -r1.95 softflowd/softflowd.c CVSWeb: http://cvsweb.mindrot.org/index.cgi/softflowd/ChangeLog?r1=1.90;r2=1.91 http://cvsweb.mindrot.org/index.cgi/softflowd/softflowd.c?r1=1.94;r2=1.95 Please note that there may be a delay before commits are available on the public CVSWeb site. From djm at fuyu.mindrot.org Wed Jul 25 09:50:35 2007 From: djm at fuyu.mindrot.org (Damien Miller) Date: Wed, 25 Jul 2007 09:50:35 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: softflowd Message-ID: <20070724235035.93C493C6B2@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: softflowd Changes by: djm at fuyu.mindrot.org 07/07/25 09:50:35 Modified files: . : ChangeLog log.c Log message: - (djm) openlog with LOG_NDELAY so socket is connected before privdrop - patch from Florian Weimer Diff commands: cvs -nQq rdiff -u -r1.91 -r1.92 softflowd/ChangeLog cvs -nQq rdiff -u -r1.2 -r1.3 softflowd/log.c CVSWeb: http://cvsweb.mindrot.org/index.cgi/softflowd/ChangeLog?r1=1.91;r2=1.92 http://cvsweb.mindrot.org/index.cgi/softflowd/log.c?r1=1.2;r2=1.3 Please note that there may be a delay before commits are available on the public CVSWeb site. From djm at fuyu.mindrot.org Thu Jul 26 10:50:31 2007 From: djm at fuyu.mindrot.org (Damien Miller) Date: Thu, 26 Jul 2007 10:50:31 +1000 (EST) Subject: [netflow-tools] CVS: fuyu.mindrot.org: softflowd Message-ID: <20070726005031.C22D53C6B2@fuyu.mindrot.org> CVSROOT: /var/cvs Module name: softflowd Changes by: djm at fuyu.mindrot.org 07/07/26 10:50:31 Modified files: . : ChangeLog softflowd.c Log message: - (djm) Add flow_get/flow_put and expiry_get/expiry_put functions to allocate and deallocate flows and expiry events, instead of calling malloc/free directly. Right now these functions just call malloc/free anyway, but they will soon be used to implemented pooled flow/expiry allocations. Diff commands: cvs -nQq rdiff -u -r1.92 -r1.93 softflowd/ChangeLog cvs -nQq rdiff -u -r1.95 -r1.96 softflowd/softflowd.c CVSWeb: http://cvsweb.mindrot.org/index.cgi/softflowd/ChangeLog?r1=1.92;r2=1.93 http://cvsweb.mindrot.org/index.cgi/softflowd/softflowd.c?r1=1.95;r2=1.96 Please note that there may be a delay before commits are available on the public CVSWeb site.