From cheeyong at asianetcom.net Wed Mar 1 20:10:18 2006 From: cheeyong at asianetcom.net (Tay Chee Yong) Date: Wed, 1 Mar 2006 17:10:18 +0800 Subject: [netflow-tools] Problem with flowd Message-ID: <006101c63d0f$f9c48ed0$6605820a@asianetcom.com> Hi, I am trying to export v9 BGP next-hop flow-aggregate to the flowd collector, and it gave me the following output when i run flowd -d # /usr/local/sbin/flowd -d read_config: entering child_get_config: entering drop_privs: dropping privs without chroot send_config: entering fd = 5 send_config: done child_get_config: child config done recv_config: entering fd = 6 recv_config: ready to receive config Listener for [127.0.0.1]:8888 fd = 4 privsep_init: entering drop_privs: dropping privs with chroot init_pfd: entering (num_fds = 0) init_pfd: done (num_fds = 2) client_open_log: entering answer_open_log: entering Continuing with existing logfile len 16 ^Cflowd_mainloop: monitor closed privsep_master: child exited My config file is : logfile "/var/log/flowd" listen on 127.0.0.1:8888 flow source store ALL discard all Can someone advice if there is any problems with my config file? Regards, Cheeyong From djm at mindrot.org Wed Mar 1 20:12:40 2006 From: djm at mindrot.org (Damien Miller) Date: Wed, 1 Mar 2006 20:12:40 +1100 (EST) Subject: [netflow-tools] Problem with flowd In-Reply-To: <006101c63d0f$f9c48ed0$6605820a@asianetcom.com> References: <006101c63d0f$f9c48ed0$6605820a@asianetcom.com> Message-ID: On Wed, 1 Mar 2006, Tay Chee Yong wrote: > Hi, > > I am trying to export v9 BGP next-hop flow-aggregate to the flowd collector, > and it gave me the following output when i run flowd -d > > # /usr/local/sbin/flowd -d > read_config: entering > child_get_config: entering > drop_privs: dropping privs without chroot > send_config: entering fd = 5 > send_config: done > child_get_config: child config done > recv_config: entering fd = 6 > recv_config: ready to receive config > Listener for [127.0.0.1]:8888 fd = 4 > privsep_init: entering > drop_privs: dropping privs with chroot > init_pfd: entering (num_fds = 0) > init_pfd: done (num_fds = 2) > client_open_log: entering > answer_open_log: entering > Continuing with existing logfile len 16 > ^Cflowd_mainloop: monitor closed > privsep_master: child exited This shows a successful startup of flowd, followed by it being manually stopped with ctrl-C. What is your exact problem? > My config file is : > > logfile "/var/log/flowd" > listen on 127.0.0.1:8888 > flow source > store ALL > discard all > > Can someone advice if there is any problems with my config file? It looks OK, except that you will never receive flows except from the localhost. Try "listen on 0.0.0.0:8888" -d From kiwi at oav.net Fri Mar 3 07:08:21 2006 From: kiwi at oav.net (Xavier Beaudouin) Date: Thu, 02 Mar 2006 21:08:21 +0100 Subject: [netflow-tools] Pfflowd and openbgpd ? Message-ID: <440750B5.7040708@oav.net> Hi there.... I am looking a way to use nfsen (http://nfsen.sf.net/) with pfflowd... But... Is there any good way to add with pfflowd source-as / dest-as using openbgpd rib ? Thanks, /Xavier From tom at public-internet.co.uk Fri Mar 3 07:22:12 2006 From: tom at public-internet.co.uk (Tom Beard) Date: Thu, 2 Mar 2006 20:22:12 +0000 Subject: [netflow-tools] Pfflowd and openbgpd ? In-Reply-To: <440750B5.7040708@oav.net> References: <440750B5.7040708@oav.net> Message-ID: <20060302202212.GA90300@honolulu.public-internet.co.uk> On Thu, Mar 02, 2006 at 09:08:21PM +0100, Xavier Beaudouin wrote: > But... Is there any good way to add with pfflowd source-as / dest-as > using openbgpd rib ? I have looked into a similar setup but didn't really have any joy, seems there is no way of getting the AS data to pfflowd. I did find mention of something about adding an extra field to OBSD's route table which could contact some misc. data such as an AS number which could then be pulled by pfflogd, however I got the impression that this was still in the theory stage. I'd be interested to know if you manage to find anything better. Regards, Tom -- Tom Beard Public Internet Limited Direct: +44 20 7993 1273 Mobile: +44 7879 817 635 From kba at ats32.ru Fri Mar 3 12:09:46 2006 From: kba at ats32.ru (kba) Date: Fri, 03 Mar 2006 09:09:46 +0800 Subject: [netflow-tools] Rotating Message-ID: <4407975A.7030009@ats32.ru> Hi everyone. What about the rotating flow records? For example flowd.2006-02-24, flowd.2006-02-25 , ...... From djm at mindrot.org Fri Mar 3 13:42:00 2006 From: djm at mindrot.org (Damien Miller) Date: Fri, 3 Mar 2006 13:42:00 +1100 (EST) Subject: [netflow-tools] Rotating In-Reply-To: <4407975A.7030009@ats32.ru> References: <4407975A.7030009@ats32.ru> Message-ID: On Fri, 3 Mar 2006, kba wrote: > Hi everyone. > What about the rotating flow records? For example flowd.2006-02-24, > flowd.2006-02-25 , ...... You can use a tool like newsyslog, logrotate, cron or a tiny shell script. Just make sure you send a SIGUSR1 or a SIGHUP to flowd after you move it. SIGUSR1 is less overhead, so it should be preferred. -d From ssnodgra at pheran.com Sat Mar 4 09:34:19 2006 From: ssnodgra at pheran.com (Steve Snodgrass) Date: Fri, 3 Mar 2006 17:34:19 -0500 Subject: [netflow-tools] Softflowd patches for ICMP type/code and DESTDIR support Message-ID: <20060303223419.GB28952@shadow.pheran.com> Greetings, First I must say thanks to Damien for this very useful program. I have recently started using softflowd and I found a few minor problems with it. 1. The Makefile doesn't support 'make install DESTDIR=' which is very useful for building RPMs (more on that in another message). I've attached a small patch that adds this support. 2. When Cisco routers generate Netflow v5 for ICMP, they encode the ICMP type and code into the Netflow destination port field as type*256 + code. Unfortunately softflowd does not do this, so you have no way of knowing what ICMP it is logging - until now! The other attached patch enables the same ICMP type/code reporting you get with Cisco Netflow. These patches are against softflowd 0.9.7. Enjoy. -- Steve Snodgrass * ssnodgra at pheran.com * Network and Unix Guru(?) at Large Geek Code: GCS d? s: a C++ U++++$ P+++ L++ w PS+ 5++ b++ DI+ D++ e++ r+++ y+* "If you want to be somebody else, change your mind." -Sister Hazel -------------- next part -------------- diff -ur softflowd-0.9.7.orig/Makefile.in softflowd-0.9.7/Makefile.in --- softflowd-0.9.7.orig/Makefile.in 2004-09-29 00:14:35.000000000 -0400 +++ softflowd-0.9.7/Makefile.in 2006-02-15 15:30:48.000000000 -0500 @@ -49,8 +49,9 @@ strip $(TARGETS) install: - $(INSTALL) -m 0755 -s softflowd $(sbindir)/softflowd - $(INSTALL) -m 0755 -s softflowctl $(sbindir)/softflowctl - $(INSTALL) -m 0644 softflowd.8 $(mandir)/man8/softflowd.8 - $(INSTALL) -m 0644 softflowctl.8 $(mandir)/man8/softflowctl.8 - + [ -d $(DESTDIR)$(sbindir) ] || mkdir -p $(DESTDIR)$(sbindir) + [ -d $(DESTDIR)$(mandir)/man8 ] || mkdir -p $(DESTDIR)$(mandir)/man8 + $(INSTALL) -m 0755 -s softflowd $(DESTDIR)$(sbindir)/softflowd + $(INSTALL) -m 0755 -s softflowctl $(DESTDIR)$(sbindir)/softflowctl + $(INSTALL) -m 0644 softflowd.8 $(DESTDIR)$(mandir)/man8/softflowd.8 + $(INSTALL) -m 0644 softflowctl.8 $(DESTDIR)$(mandir)/man8/softflowctl.8 -------------- next part -------------- diff -ur softflowd-0.9.7.orig/common.h softflowd-0.9.7/common.h --- softflowd-0.9.7.orig/common.h 2005-01-14 23:08:56.000000000 -0500 +++ softflowd-0.9.7/common.h 2006-03-03 15:23:30.000000000 -0500 @@ -41,6 +41,7 @@ #include #include #include +#include #include #include #include diff -ur softflowd-0.9.7.orig/softflowd.c softflowd-0.9.7/softflowd.c --- softflowd-0.9.7.orig/softflowd.c 2005-01-09 20:50:07.000000000 -0500 +++ softflowd-0.9.7/softflowd.c 2006-03-03 16:36:44.000000000 -0500 @@ -282,6 +282,7 @@ { const struct tcphdr *tcp = (const struct tcphdr *)pkt; const struct udphdr *udp = (const struct udphdr *)pkt; + const struct icmphdr *icmp = (const struct icmphdr *)pkt; /* * XXX to keep flow in proper canonical format, it may be necessary @@ -306,6 +307,11 @@ flow->port[ndx] = udp->uh_sport; flow->port[ndx ^ 1] = udp->uh_dport; break; + case IPPROTO_ICMP: + /* Encode ICMP type * 256 + code into dest port like Cisco routers */ + flow->port[ndx] = 0; + flow->port[ndx ^ 1] = htons(icmp->type * 256 + icmp->code); + break; } return (0); } From ssnodgra at pheran.com Sat Mar 4 09:55:58 2006 From: ssnodgra at pheran.com (Steve Snodgrass) Date: Fri, 3 Mar 2006 17:55:58 -0500 Subject: [netflow-tools] Building a softflowd RPM Message-ID: <20060303225558.GA29072@shadow.pheran.com> I've included several files that enable one to build an RPM package of softflowd. This is intended for Red Hat (i.e. Fedora Core and RHEL) systems but could be ported to any RPM-based system without too much trouble. The attached files are: softflowd.spec - the RPM spec file softflowd.init - an init script for starting/stopping softflowd softflowd.sysconfig - a config file that sets options for softflowd This RPM also makes use of the DESTDIR and ICMP patches I submitted in my previous email. The ICMP patch is optional and could be removed, but the DESTDIR patch is needed to successfully build the RPM. I could make a full source RPM available if there's demand for it. -- Steve Snodgrass * ssnodgra at pheran.com * Network and Unix Guru(?) at Large Geek Code: GCS d? s: a C++ U++++$ P+++ L++ w PS+ 5++ b++ DI+ D++ e++ r+++ y+* "If you want to be somebody else, change your mind." -Sister Hazel -------------- next part -------------- # $Id: softflowd.spec,v 1.4 2006/03/03 21:39:09 ssnodgra_dev Exp $ # Figure out release tag (e.g. rhel3, fc1) based on redhat-release file %global _RHTAG %(sed 's/(.*)//;s/ [A-Z]* //;s/[a-z ]*//g' /etc/redhat-release | tr '[:upper:]' '[:lower:]') Name: softflowd Summary: Network traffic analyser capable of Cisco NetFlow data export Version: 0.9.7 Release: 4.%{_RHTAG} Source: softflowd-%{version}.tar.gz Group: System/Utilities License: BSD BuildRoot: %{_tmppath}/%{name}-root URL: http://www.mindrot.org/softflowd.html Vendor: mindrot.org Patch0: destdir.patch Patch1: icmp.patch %description softflowd is a software implementation of a flow-based network traffic monitor. softflowd reads network traffic and gathers information about active traffic flows. A "traffic flow" is communication between two IP addresses or (if the overlying protocol is TCP or UDP) address/port tuples. The intended use of softflowd is as a software implementation of Cisco?s NetFlow traffic account system. softflowd supports data export using versions 1, 5 or 9 of the NetFlow protocol. %prep %setup %patch0 -p1 %patch1 -p1 %build %configure make %install rm -rf $RPM_BUILD_ROOT make install DESTDIR=$RPM_BUILD_ROOT mkdir -p $RPM_BUILD_ROOT/etc/rc.d/init.d mkdir -p $RPM_BUILD_ROOT/etc/sysconfig cp %_sourcedir/softflowd.init $RPM_BUILD_ROOT/etc/rc.d/init.d/softflowd cp %_sourcedir/softflowd.sysconfig $RPM_BUILD_ROOT/etc/sysconfig/softflowd %files %defattr(-,root,root) /usr/sbin/* /usr/share/man/* %attr(0755,root,root) /etc/rc.d/init.d/softflowd %config(noreplace) %attr(0644,root,root) /etc/sysconfig/softflowd %doc ChangeLog README TODO %clean rm -rf $RPM_BUILD_ROOT -------------- next part -------------- #!/bin/bash # # softflowd Starts softflowd NetFlow probe # $Id: softflowd.init,v 1.2 2006/02/15 21:42:04 ssnodgra_dev Exp $ # # chkconfig: 2345 95 02 # description: Starts and stops the softflowd Netflow probe # Source function library. . /etc/init.d/functions SOFTFLOW_CONF=/etc/sysconfig/softflowd SOFTFLOW_LOCK=/var/lock/subsys/softflowd SOFTFLOW_PROG=/usr/sbin/softflowd SOFTFLOW_OPTS="-i eth0" # Source config if [ -f $SOFTFLOW_CONF ]; then . $SOFTFLOW_CONF fi [ -x $SOFTFLOW_PROG ] || exit 0 RETVAL=0 start() { echo -n $"Starting softflowd: " daemon $SOFTFLOW_PROG $SOFTFLOW_OPTS RETVAL=$? echo [ $RETVAL -eq 0 ] && touch $SOFTFLOW_LOCK return $RETVAL } stop() { echo -n $"Shutting down softflowd: " killproc $SOFTFLOW_PROG RETVAL=$? echo [ $RETVAL -eq 0 ] && rm -f $SOFTFLOW_LOCK return $RETVAL } restart() { stop start } case "$1" in start) start ;; stop) stop ;; status) status $SOFTFLOW_PROG ;; restart|reload) restart ;; condrestart) [ -f $SOFTFLOW_LOCK ] && restart || : ;; *) echo $"Usage: $0 {start|stop|status|restart|condrestart}" exit 1 esac exit $? -------------- next part -------------- # Config file for softflowd startup # Location of softflowd binary #SOFTFLOW_PROG=/usr/sbin/softflowd # Options passed to the softflowd program # Default - not very useful #SOFTFLOW_OPTS="-i eth0" # Example NetFlow v5 export from traffic on eth1 #SOFTFLOW_OPTS="-v 5 -i eth1 -n hostname:2055" From djm at mindrot.org Sat Mar 4 15:57:33 2006 From: djm at mindrot.org (Damien Miller) Date: Sat, 4 Mar 2006 15:57:33 +1100 (EST) Subject: [netflow-tools] Summarisation and charting of flowd logs Message-ID: Hi, Attached is something I cooked up this morning: a basic tool to summarise flowd logs and store the results in an RRD[1] database, and a shell script to generate charts of the results. An example of the output is at: http://www.mindrot.org/~djm/tmp/flows.html (note that this URL will change once I hook a better example up to the flowd project page, and tidy this mail into a HOWTO). Right now it is pretty basic: just a basic protocol breakdown across TCP, UDP, ICMP, GRE, ESP IPsec and "other". However, the tools can be easily extended to summarise pretty much anything you can express in Python. Another mode of summarisation I am considering adding is clustering flows by the "tag" that can be set in a flowd.conf filter - so setting up summary classes is just a matter of writing a filter specification. Please let me know what would work best for you. Note that the main script ("flow_rrd.py") requires py-rrdtool[2]. It is able to read either historic flowd logs or listen for live updates using flowd-0.9's experimental logging socket support. The first mode (historical logs) is likely to be more stable; it works like this: ./flow_rrd.py /path/to/flowd_log /path/to/database.rrd If the RRD database doesn't already exist, then it will be created with some reasonable defaults automatically. The database will be populated with summaries of whatever flows were in the log file. Once the RRD database has been created and filled in, it can be graphed using the "plot.sh" script, like this: ./plot.sh /path/to/database.rrd /output/directory/for/images/ This script will create daily, weekly, monthly and yearly images showing flows-per-second, bytes-per-second and packets-per-second. The HTML files in the archive/CVS display these all on one page. In real use, the summarisation should probably be hooked up to occur when the log is rotated and the plotting should be run afterwards or out of cron. Note that this accounts flows to the period when they were *received*, so traffic flows lasting greater than five minutes will cause spikes in the chart. To work around this, you can instruct your flow exported to expire flows at the five minute mark. For softflowd[1], add the command-line argument "-tmaxlife=300". On a recent-ish Cisco device, use the command: ip flow-cache timeout active 5 These tools have just been committed to CVS, so they will also show up in the snapshot releases. They likely still have a few bugs, and I would appreciate any testing list lurkers are able to offer. Cheers, Damien Miller -------------- next part -------------- A non-text attachment was scrubbed... Name: flowrrd.tgz Type: application/octet-stream Size: 3933 bytes Desc: Url : http://lists.mindrot.org/pipermail/netflow-tools/attachments/20060304/0b2ef291/attachment.obj From fw at deneb.enyo.de Sun Mar 12 21:23:52 2006 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 12 Mar 2006 11:23:52 +0100 Subject: [netflow-tools] Summarisation and charting of flowd logs In-Reply-To: (Damien Miller's message of "Sat, 4 Mar 2006 15:57:33 +1100 (EST)") References: Message-ID: <87psksotaf.fsf@mid.deneb.enyo.de> * Damien Miller: > Attached is something I cooked up this morning: a basic tool to > summarise flowd logs and store the results in an RRD[1] database, > and a shell script to generate charts of the results. There is also a tool called NfSen: From ssnodgra at pheran.com Mon Mar 13 03:05:46 2006 From: ssnodgra at pheran.com (Steve Snodgrass) Date: Sun, 12 Mar 2006 11:05:46 -0500 Subject: [netflow-tools] Summarisation and charting of flowd logs In-Reply-To: <87psksotaf.fsf@mid.deneb.enyo.de> References: <87psksotaf.fsf@mid.deneb.enyo.de> Message-ID: <20060312160546.GB3309@shadow.pheran.com> On Sun, Mar 12, 2006 at 11:23:52AM +0100, Florian Weimer wrote: > * Damien Miller: > > > Attached is something I cooked up this morning: a basic tool to > > summarise flowd logs and store the results in an RRD[1] database, > > and a shell script to generate charts of the results. > > There is also a tool called NfSen: Yes, in fact I'm using softflowd to send flows to nfdump (and I will soon add NfSen on top of that). -- Steve Snodgrass * ssnodgra at pheran.com * Network and Unix Guru(?) at Large Geek Code: GCS d? s: a C++ U++++$ P+++ L++ w PS+ 5++ b++ DI+ D++ e++ r+++ y+* "If you want to be somebody else, change your mind." -Sister Hazel From djm at mindrot.org Tue Mar 14 16:54:55 2006 From: djm at mindrot.org (Damien Miller) Date: Tue, 14 Mar 2006 16:54:55 +1100 (EST) Subject: [netflow-tools] Pfflowd and openbgpd ? In-Reply-To: <20060302202212.GA90300@honolulu.public-internet.co.uk> References: <440750B5.7040708@oav.net> <20060302202212.GA90300@honolulu.public-internet.co.uk> Message-ID: On Thu, 2 Mar 2006, Tom Beard wrote: > On Thu, Mar 02, 2006 at 09:08:21PM +0100, Xavier Beaudouin wrote: > > But... Is there any good way to add with pfflowd source-as / dest-as > > using openbgpd rib ? > > I have looked into a similar setup but didn't really have any joy, > seems there is no way of getting the AS data to pfflowd. I did find > mention of something about adding an extra field to OBSD's route table > which could contact some misc. data such as an AS number which could > then be pulled by pfflogd, however I got the impression that this was > still in the theory stage. It should be possible to let pfflowd look up the AS of an address using bgpd's looking glass read-only socket. I haven't looked at it yet, but it wouldn't be too much work. -d From nathan at inorb.com Wed Mar 15 06:44:17 2006 From: nathan at inorb.com (Nathan Einwechter) Date: Tue, 14 Mar 2006 14:44:17 -0500 Subject: [netflow-tools] Flowd Filter Question Message-ID: <001701c6479f$ae404ea0$6400a8c0@DHRWZP71> I just installed flowd as part of a security management system I'm trying to pull together and am trying to refine the collection of NetFlow logs to reduce the amount of space eaten by the logs. As such, I am trying to filter out those entries I'm not interested in. Specifically, I am trying to filter out (discard) anything non-UDP or TCP and any connection which was not established (obviously for TCP only, we'll keep all UDP). How can this be done? I've been fiddling with the filters for a couple days now and just can't seem to get it. Thanks for any help and take care everyone! Yours truly, Nathan From djm at mindrot.org Wed Mar 15 07:39:02 2006 From: djm at mindrot.org (Damien Miller) Date: Wed, 15 Mar 2006 07:39:02 +1100 (EST) Subject: [netflow-tools] Flowd Filter Question In-Reply-To: <001701c6479f$ae404ea0$6400a8c0@DHRWZP71> References: <001701c6479f$ae404ea0$6400a8c0@DHRWZP71> Message-ID: On Tue, 14 Mar 2006, Nathan Einwechter wrote: > I just installed flowd as part of a security management system I'm > trying to pull together and am trying to refine the collection of > NetFlow logs to reduce the amount of space eaten by the logs. As such, I > am trying to filter out those entries I'm not interested in. > Specifically, I am trying to filter out (discard) anything non-UDP or > TCP and any connection which was not established (obviously for TCP > only, we'll keep all UDP). > > How can this be done? I've been fiddling with the filters for a couple > days now and just can't seem to get it. You should be able to do something like: discard all accept proto udp accept proto tcp tcp_flags mask 0x12 equals 0x12 # ACK = 0x10, SYN = 0x02 -d From djm at mindrot.org Wed Mar 15 10:09:57 2006 From: djm at mindrot.org (Damien Miller) Date: Wed, 15 Mar 2006 10:09:57 +1100 (EST) Subject: [netflow-tools] Softflowd patches for ICMP type/code and DESTDIR support In-Reply-To: <20060303223419.GB28952@shadow.pheran.com> References: <20060303223419.GB28952@shadow.pheran.com> Message-ID: On Fri, 3 Mar 2006, Steve Snodgrass wrote: > Greetings, > > First I must say thanks to Damien for this very useful program. I have > recently started using softflowd and I found a few minor problems with it. Thanks! > 1. The Makefile doesn't support 'make install DESTDIR=' which is very > useful for building RPMs (more on that in another message). I've attached > a small patch that adds this support. Applied. > 2. When Cisco routers generate Netflow v5 for ICMP, they encode the ICMP > type and code into the Netflow destination port field as type*256 + code. > Unfortunately softflowd does not do this, so you have no way of knowing > what ICMP it is logging - until now! The other attached patch enables > the same ICMP type/code reporting you get with Cisco Netflow. Thanks for this. I tweaked the patch slightly because "struct icmphdr" appears to be a Linuxism, and is not present on OpenBSD or Solaris. What was committed uses "struct icmp" which is everywhere. Please give this a try - it might need some incantation of _BSD_SOURCE defined on glibc, or maybe not. -d Index: common.h =================================================================== RCS file: /var/cvs/softflowd/common.h,v retrieving revision 1.22 diff -u -p -r1.22 common.h --- common.h 15 Jan 2005 04:08:56 -0000 1.22 +++ common.h 14 Mar 2006 22:56:16 -0000 @@ -41,6 +41,7 @@ #include #include #include +#include #include #include #include Index: softflowd.c =================================================================== RCS file: /var/cvs/softflowd/softflowd.c,v retrieving revision 1.90 diff -u -p -r1.90 softflowd.c --- softflowd.c 14 Mar 2006 22:51:48 -0000 1.90 +++ softflowd.c 14 Mar 2006 23:04:09 -0000 @@ -285,6 +285,7 @@ transport_to_flowrec(struct FLOW *flow, { const struct tcphdr *tcp = (const struct tcphdr *)pkt; const struct udphdr *udp = (const struct udphdr *)pkt; + const struct icmp *icmp = (const struct icmp *)pkt; /* * XXX to keep flow in proper canonical format, it may be necessary @@ -308,6 +309,15 @@ transport_to_flowrec(struct FLOW *flow, return (isfrag ? 0 : 1); flow->port[ndx] = udp->uh_sport; flow->port[ndx ^ 1] = udp->uh_dport; + break; + case IPPROTO_ICMP: + /* + * Encode ICMP type * 256 + code into dest port like + * Cisco routers + */ + flow->port[ndx] = 0; + flow->port[ndx ^ 1] = htons(icmp->icmp_type * 256 + + icmp->icmp_code); break; } return (0); From djm at mindrot.org Wed Mar 15 10:17:20 2006 From: djm at mindrot.org (Damien Miller) Date: Wed, 15 Mar 2006 10:17:20 +1100 (EST) Subject: [netflow-tools] Building a softflowd RPM In-Reply-To: <20060303225558.GA29072@shadow.pheran.com> References: <20060303225558.GA29072@shadow.pheran.com> Message-ID: On Fri, 3 Mar 2006, Steve Snodgrass wrote: > I've included several files that enable one to build an RPM package of > softflowd. This is intended for Red Hat (i.e. Fedora Core and RHEL) systems > but could be ported to any RPM-based system without too much trouble. Thanks again, I have committed these so they will show up in the snapshots (http://www2.mindrot.org/softflowd_snap/) tonight. The snapshots now have a version number of 0.9.8, and I will probably do a new release soon - there have been quite a few little fixes since 0.9.7 was released. Thanks, Damien Miller From nathan at inorb.com Thu Mar 16 03:41:16 2006 From: nathan at inorb.com (Nathan Einwechter) Date: Wed, 15 Mar 2006 11:41:16 -0500 Subject: [netflow-tools] Flowd Perl Examples Message-ID: <000e01c6484f$479e49d0$6400a8c0@DHRWZP71> I was just curious as to whether there are any examples out there for the usage of the Flowd Perl interface. I searched online but did not find anything. Thanks for everyone's assistance. -- Nathan Einwechter From djm at mindrot.org Thu Mar 16 07:43:03 2006 From: djm at mindrot.org (Damien Miller) Date: Thu, 16 Mar 2006 07:43:03 +1100 (EST) Subject: [netflow-tools] Flowd Perl Examples In-Reply-To: <000e01c6484f$479e49d0$6400a8c0@DHRWZP71> References: <000e01c6484f$479e49d0$6400a8c0@DHRWZP71> Message-ID: On Wed, 15 Mar 2006, Nathan Einwechter wrote: > I was just curious as to whether there are any examples out there for > the usage of the Flowd Perl interface. I searched online but did not > find anything. There is tools/flowinsert.pl in the source distributions, a small program to stuff flows into a SQLite database. -d From ssnodgra at pheran.com Thu Mar 16 11:56:52 2006 From: ssnodgra at pheran.com (Steve Snodgrass) Date: Wed, 15 Mar 2006 19:56:52 -0500 Subject: [netflow-tools] Pfflowd and openbgpd ? In-Reply-To: References: <440750B5.7040708@oav.net> <20060302202212.GA90300@honolulu.public-internet.co.uk> Message-ID: <20060316005652.GA11525@shadow.pheran.com> On Tue, Mar 14, 2006 at 04:54:55PM +1100, Damien Miller wrote: > It should be possible to let pfflowd look up the AS of an address > using bgpd's looking glass read-only socket. I haven't looked at it > yet, but it wouldn't be too much work. It may be even easier to use a DNS lookup with Cymru's IP to ASN service. http://www.cymru.com/BGP/asnlookup.html -- Steve Snodgrass * ssnodgra at pheran.com * Network and Unix Guru(?) at Large Geek Code: GCS d? s: a C++ U++++$ P+++ L++ w PS+ 5++ b++ DI+ D++ e++ r+++ y+* "If you want to be somebody else, change your mind." -Sister Hazel From murray.shields at netoptions.com.au Fri Mar 24 13:47:23 2006 From: murray.shields at netoptions.com.au (Murray Shields) Date: Fri, 24 Mar 2006 12:47:23 +1000 Subject: [netflow-tools] flowd-reader export Message-ID: <44235DBB.1040401@netoptions.com.au> Is there any documentation on the export as generated by flowd-reader? For example, what are the possible values and meanings for proto (I know 6 is TCP)? What is the most accurate way of matching bi-directional packets (is it simply a specific port number range)? Can I simply assume that the LOWER port number is the port, and the higher is for matching? I have tried all of the README files, installed documentation and Googled, but can find nothing on this. I have also grepped a downloaded copy of the mailing list archive. Can anyone help? Thanks. Murray. From gijs at looze.net Fri Mar 24 18:52:11 2006 From: gijs at looze.net (Gijs Molenaar) Date: Fri, 24 Mar 2006 08:52:11 +0100 Subject: [netflow-tools] flowd-reader export In-Reply-To: <44235DBB.1040401@netoptions.com.au> References: <44235DBB.1040401@netoptions.com.au> Message-ID: <4423A52B.3050806@looze.net> Murray Shields schreef: > Is there any documentation on the export as generated by flowd-reader? > For example, what are the possible values and meanings for proto (I know > 6 is TCP)? http://www.iana.org/assignments/protocol-numbers googlin for 'ip protocol numbers' was quite usefull. > What is the most accurate way of matching bi-directional > packets (is it simply a specific port number range)? > Can I simply assume that the LOWER port number is the port, and the > higher is for matching? > By my knowledge flows are uni-directional. So if you have a TCP session, 2 flows are created. There is a source and destination port, but now lower and higher. But maybe I'm wrong... From jp.luiggi at free.fr Sat Mar 25 02:14:37 2006 From: jp.luiggi at free.fr (Jean-Philippe Luiggi) Date: Fri, 24 Mar 2006 10:14:37 -0500 Subject: [netflow-tools] flowd-reader export In-Reply-To: <44235DBB.1040401@netoptions.com.au> References: <44235DBB.1040401@netoptions.com.au> Message-ID: <20060324151437.GB30567@armada.mynetwork.net> Hello, On Fri, Mar 24, 2006 at 12:47:23PM +1000, Murray Shields wrote: > > Is there any documentation on the export as generated by flowd-reader? > For example, what are the possible values and meanings for proto (I know > 6 is TCP)? What is the most accurate way of matching bi-directional > packets (is it simply a specific port number range)? About protocols : less /etc/protocols (Unix) or "www.iana.org" and for "bi-directional matching": on the server's side, there's one defined port but from client's point of vue, it's not true. > Can I simply assume that the LOWER port number is the port, and the > higher is for matching? I'm not sure to understand what do you want to say ? Best regards. From fw at deneb.enyo.de Sat Mar 25 08:03:12 2006 From: fw at deneb.enyo.de (Florian Weimer) Date: Fri, 24 Mar 2006 22:03:12 +0100 Subject: [netflow-tools] flowd-reader export In-Reply-To: <44235DBB.1040401@netoptions.com.au> (Murray Shields's message of "Fri, 24 Mar 2006 12:47:23 +1000") References: <44235DBB.1040401@netoptions.com.au> Message-ID: <87r74rimi7.fsf@mid.deneb.enyo.de> * Murray Shields: > Is there any documentation on the export as generated by flowd-reader? > For example, what are the possible values and meanings for proto (I know > 6 is TCP)? What is the most accurate way of matching bi-directional > packets (is it simply a specific port number range)? You can match the connection quadruple (twice IP address and port). They are the same for both directions, except that sender and receiver are swapped. From nathan at inorb.com Sat Mar 25 11:04:47 2006 From: nathan at inorb.com (Nathan Einwechter) Date: Fri, 24 Mar 2006 19:04:47 -0500 Subject: [netflow-tools] flowd-reader export In-Reply-To: <44235DBB.1040401@netoptions.com.au> Message-ID: <019901c64f9f$be3979f0$0101080a@DHRWZP71> Along the same lines of this question; when NetFlow lists something as being the "Source", for TCP connections, does this mean the full connection source (within the context of a TCP connection, three-way-handshake etc), or just where that specific set of packets is going to/coming from? i.e. if I'm looking at web traffic, will it look like this Source Dest SrcPort DstPort Prot A B 1064 80 6 Or this: Source Dest SrcPort DstPort Prot A B 1064 80 6 B A 80 1064 6 ? Thanks for everyone's assistance in clarifying this. Yours truly, Nathan -----Original Message----- From: netflow-tools-bounces+nathan=inorb.com at mindrot.org [mailto:netflow-tools-bounces+nathan=inorb.com at mindrot.org] On Behalf Of Murray Shields Sent: March 23, 2006 9:47 PM To: netflow-tools at mindrot.org Subject: [netflow-tools] flowd-reader export Is there any documentation on the export as generated by flowd-reader? For example, what are the possible values and meanings for proto (I know 6 is TCP)? What is the most accurate way of matching bi-directional packets (is it simply a specific port number range)? Can I simply assume that the LOWER port number is the port, and the higher is for matching? I have tried all of the README files, installed documentation and Googled, but can find nothing on this. I have also grepped a downloaded copy of the mailing list archive. Can anyone help? Thanks. Murray. _______________________________________________ netflow-tools mailing list netflow-tools at mindrot.org http://www.mindrot.org/mailman/listinfo/netflow-tools From djm at mindrot.org Sat Mar 25 11:12:54 2006 From: djm at mindrot.org (Damien Miller) Date: Sat, 25 Mar 2006 11:12:54 +1100 (EST) Subject: [netflow-tools] flowd-reader export In-Reply-To: <019901c64f9f$be3979f0$0101080a@DHRWZP71> References: <019901c64f9f$be3979f0$0101080a@DHRWZP71> Message-ID: On Fri, 24 Mar 2006, Nathan Einwechter wrote: > > Along the same lines of this question; when NetFlow lists something as > being the "Source", for TCP connections, does this mean the full > connection source (within the context of a TCP connection, > three-way-handshake etc), or just where that specific set of packets is > going to/coming from? The latter, unfortunately. NetFlow's design shows its lineage as part of Cisco's old forwarding cache - it doesn't have any conceptions of bidirectionality. Even NetFlow v.9 has not addressed this problem. Maybe IPFIX (IETF flow export) will, but I haven't looked at the drafts for a while. -d From murray.shields at netoptions.com.au Mon Mar 27 11:00:19 2006 From: murray.shields at netoptions.com.au (Murray Shields) Date: Mon, 27 Mar 2006 10:00:19 +1000 Subject: [netflow-tools] flowd-reader export In-Reply-To: <4423A52B.3050806@looze.net> References: <44235DBB.1040401@netoptions.com.au> <4423A52B.3050806@looze.net> Message-ID: <44272B13.4020008@netoptions.com.au> Gijs Molenaar wrote: > Murray Shields schreef: >> Is there any documentation on the export as generated by >> flowd-reader? For example, what are the possible values and meanings >> for proto (I know 6 is TCP)? > http://www.iana.org/assignments/protocol-numbers > > googlin for 'ip protocol numbers' was quite usefull. Excellent, thank you. >> What is the most accurate way of matching bi-directional packets (is >> it simply a specific port number range)? >> Can I simply assume that the LOWER port number is the port, and the >> higher is for matching? >> > By my knowledge flows are uni-directional. So if you have a TCP > session, 2 flows are > created. There is a source and destination port, but now lower and > higher. But maybe > I'm wrong... > From murray.shields at netoptions.com.au Mon Mar 27 11:14:33 2006 From: murray.shields at netoptions.com.au (Murray Shields) Date: Mon, 27 Mar 2006 10:14:33 +1000 Subject: [netflow-tools] flowd-reader export In-Reply-To: <20060324151437.GB30567@armada.mynetwork.net> References: <44235DBB.1040401@netoptions.com.au> <20060324151437.GB30567@armada.mynetwork.net> Message-ID: <44272E69.7040404@netoptions.com.au> Jean-Philippe Luiggi wrote: > Hello, > > On Fri, Mar 24, 2006 at 12:47:23PM +1000, Murray Shields wrote: > >> Is there any documentation on the export as generated by flowd-reader? >> For example, what are the possible values and meanings for proto (I know >> 6 is TCP)? What is the most accurate way of matching bi-directional >> packets (is it simply a specific port number range)? >> > > About protocols : less /etc/protocols (Unix) or "www.iana.org" > and for "bi-directional matching": on the server's side, there's one defined > port but from client's point of vue, it's not true. > > >> Can I simply assume that the LOWER port number is the port, and the >> higher is for matching? >> > > I'm not sure to understand what do you want to say ? > UDP packets are easy: in the test feed I am currently looking at the source has a port of zero (0) and the destination 771. For example: [192.168.1.1]:0 => [192.168.2.254]:771 But for unidirectional we are effectively getting two port numbers via two matching flow records. For example: [192.168.2.1]:45223 => [192.168.1.1]:80 [192.168.1.1]:80 => [192.168.2.1]:45223 For my purposes I do not need to match these two lines as this is for an ISP billing system (and they only bill for outgoing, not incoming). I also need to allocate the traffic to services via the ip address (eg, group port 80 web traffic on the bill). Looking at a single line in isolation the other port (45223) is significantly higher that the port for web traffic (80) that I am interested in. So to rephrase the question: can I assume that the port number that I need to look at is ALWAYS the lower of the two port numbers? That is, discard the higher number (45223) and assume that the traffic relates to the lower port (in this instance, port 80)? Is there a circumstance where this would fail me? Regards, Murray. > Best regards. > > From murray.shields at netoptions.com.au Mon Mar 27 14:51:51 2006 From: murray.shields at netoptions.com.au (Murray Shields) Date: Mon, 27 Mar 2006 13:51:51 +1000 Subject: [netflow-tools] flowd-reader export In-Reply-To: <87r74rimi7.fsf@mid.deneb.enyo.de> References: <44235DBB.1040401@netoptions.com.au> <87r74rimi7.fsf@mid.deneb.enyo.de> Message-ID: <44276157.4040508@netoptions.com.au> Florian Weimer wrote: > * Murray Shields: > > >> Is there any documentation on the export as generated by flowd-reader? >> For example, what are the possible values and meanings for proto (I know >> 6 is TCP)? What is the most accurate way of matching bi-directional >> packets (is it simply a specific port number range)? >> > > You can match the connection quadruple (twice IP address and port). > They are the same for both directions, except that sender and receiver > are swapped. > When you perform this match (I will have to add the received time into this equation as I am getting repeats at different times) this will give me a bi-directional pair for a request/response flow of traffic. THEREFORE can I use the destination port from the FIRST of these two records, and use it as the port identifying the type of traffic? For instance, the following matched pair: [192.168.2.1]:45223 => [192.168.1.1]:80 [192.168.1.1]:80 => [192.168.2.1]:45223 means: 192.168.2.1 used port 54223 to send a packet request a web server on 192.168.1.1 using port 80. 192.168.1.1 used port 80 to send a response to 192.168.2.1 using port 54223. Therefore the port indicating the traffic type is 80 (the first destination). Makes sense to me. Any holes in this logic? Murray. From fw at deneb.enyo.de Tue Mar 28 23:00:50 2006 From: fw at deneb.enyo.de (Florian Weimer) Date: Tue, 28 Mar 2006 14:00:50 +0200 Subject: [netflow-tools] flowd-reader export In-Reply-To: <44276157.4040508@netoptions.com.au> (Murray Shields's message of "Mon, 27 Mar 2006 13:51:51 +1000") References: <44235DBB.1040401@netoptions.com.au> <87r74rimi7.fsf@mid.deneb.enyo.de> <44276157.4040508@netoptions.com.au> Message-ID: <87hd5ipyml.fsf@mid.deneb.enyo.de> * Murray Shields: > Makes sense to me. Any holes in this logic? It might be a very long connection which results in multiple flows. In this case, the first packet in the two flows is not sent by the client. In general, it is quite difficult to reconstruct the roles without TCP flags export (and the way it is done by some vendors is not really helpful, either). From yb at bashibuzuk.net Wed Mar 29 00:45:51 2006 From: yb at bashibuzuk.net (Yann Berthier) Date: Tue, 28 Mar 2006 15:45:51 +0200 Subject: [netflow-tools] flowd-reader export In-Reply-To: <87hd5ipyml.fsf@mid.deneb.enyo.de> References: <44235DBB.1040401@netoptions.com.au> <87r74rimi7.fsf@mid.deneb.enyo.de> <44276157.4040508@netoptions.com.au> <87hd5ipyml.fsf@mid.deneb.enyo.de> Message-ID: <20060328134551.GG1122@bashibuzuk.net> On Tue, 28 Mar 2006, at 14:00, Florian Weimer wrote: > * Murray Shields: > > > Makes sense to me. Any holes in this logic? > > It might be a very long connection which results in multiple flows. > In this case, the first packet in the two flows is not sent by the > client. > > > In general, it is quite difficult to reconstruct the roles without TCP > flags export (and the way it is done by some vendors is not really > helpful, either). Even when you are lucky enough to have the flags, it not that helpful: as flags are ORed, you end up for a 'complete' tcp 'session' with both uni-directional flows having at least SAF set - no way to distinguish the client (in an ip sense) from the server Or do i minsunderstand you ? From yb at bashibuzuk.net Wed Mar 29 00:56:32 2006 From: yb at bashibuzuk.net (Yann Berthier) Date: Tue, 28 Mar 2006 15:56:32 +0200 Subject: [netflow-tools] flowd-reader export In-Reply-To: References: <019901c64f9f$be3979f0$0101080a@DHRWZP71> Message-ID: <20060328135632.GH1122@bashibuzuk.net> On Sat, 25 Mar 2006, at 11:12, Damien Miller wrote: > On Fri, 24 Mar 2006, Nathan Einwechter wrote: > > > > > Along the same lines of this question; when NetFlow lists something as > > being the "Source", for TCP connections, does this mean the full > > connection source (within the context of a TCP connection, > > three-way-handshake etc), or just where that specific set of packets is > > going to/coming from? > > The latter, unfortunately. > > NetFlow's design shows its lineage as part of Cisco's old forwarding > cache - it doesn't have any conceptions of bidirectionality. Even > NetFlow v.9 has not addressed this problem. > > Maybe IPFIX (IETF flow export) will, but I haven't looked at the > drafts for a while. There is a biflow draft btw: http://www.ietf.org/internet-drafts/draft-boschi-ipfix-biflow-01.txt there is discussions regarding this draft on the ipfix list From yb at bashibuzuk.net Wed Mar 29 02:00:27 2006 From: yb at bashibuzuk.net (Yann Berthier) Date: Tue, 28 Mar 2006 17:00:27 +0200 Subject: [netflow-tools] flowd-reader export In-Reply-To: <20060328135632.GH1122@bashibuzuk.net> References: <019901c64f9f$be3979f0$0101080a@DHRWZP71> <20060328135632.GH1122@bashibuzuk.net> Message-ID: <20060328150027.GL1122@bashibuzuk.net> On Tue, 28 Mar 2006, at 15:56, Yann Berthier wrote: > There is a biflow draft btw: > http://www.ietf.org/internet-drafts/draft-boschi-ipfix-biflow-01.txt Sorry, should have been http://www.ietf.org/internet-drafts/draft-trammell-ipfix-biflow-00.txt From nathan at inorb.com Wed Mar 29 03:15:53 2006 From: nathan at inorb.com (Nathan Einwechter) Date: Tue, 28 Mar 2006 11:15:53 -0500 Subject: [netflow-tools] flowd-reader export In-Reply-To: <20060328134551.GG1122@bashibuzuk.net> Message-ID: <00b601c65282$e32e8260$0101080a@DHRWZP71> Yann Said: Even when you are lucky enough to have the flags, it not that helpful: as flags are ORed, you end up for a 'complete' tcp 'session' with both uni-directional flows having at least SAF set - no way to distinguish the client (in an ip sense) from the server Or do i minsunderstand you ---------------------- Okay - here's what I'm doing now as a test and want to see if this will work as I anticipate. For TCP connections, I'm filtering only those that are active connections (in my case, I don't care about those that aren't full fledged connections) using the flags ala: tcp_flags mask 0x12 equals 0x12 This creates a situation where the true connection source and destinations are reversed in the log, due to the stage of communications that these flags are set. So, when I export them using my perl exporter, I simply invert them once again to get the true source and destination for my final processing. Does this work as I anticipate? Would this give me the actual source and destinations? From what I've seen it does, but there may be exceptions. Also, you mention, in a later message, that connections separated by significant time will not be aggregated into a single entry. Any idea how long this is etc? That becomes important. I have a long and memory intensive process to remove these duplicates, but if I could have a timeframe after which duplicate entries are not inserted, then I could reduce the inefficiency of this process. Thanks again. -- Nathan From ssnodgra at pheran.com Wed Mar 29 04:53:47 2006 From: ssnodgra at pheran.com (Steve Snodgrass) Date: Tue, 28 Mar 2006 12:53:47 -0500 Subject: [netflow-tools] Softflowd patches for ICMP type/code and DESTDIR support In-Reply-To: References: <20060303223419.GB28952@shadow.pheran.com> Message-ID: <20060328175347.GA24899@shadow.pheran.com> On Wed, Mar 15, 2006 at 10:09:57AM +1100, Damien Miller wrote: > > 2. When Cisco routers generate Netflow v5 for ICMP, they encode the ICMP > > type and code into the Netflow destination port field as type*256 + code. > > Unfortunately softflowd does not do this, so you have no way of knowing > > what ICMP it is logging - until now! The other attached patch enables > > the same ICMP type/code reporting you get with Cisco Netflow. > > Thanks for this. I tweaked the patch slightly because "struct icmphdr" > appears to be a Linuxism, and is not present on OpenBSD or Solaris. > What was committed uses "struct icmp" which is everywhere. > > Please give this a try - it might need some incantation of _BSD_SOURCE > defined on glibc, or maybe not. You're absolutely right, sorry about that. I just checked an old Solaris 8 box and it only has "struct icmp" as well. I did a compile on Linux with struct icmp and it worked fine (with no additional defines), so your patch should be good. -- Steve Snodgrass * ssnodgra at pheran.com * Network and Unix Guru(?) at Large Geek Code: GCS d? s: a C++ U++++$ P+++ L++ w PS+ 5++ b++ DI+ D++ e++ r+++ y+* "If you want to be somebody else, change your mind." -Sister Hazel