From andreas.brillisauer at hetzner.de Wed Feb 1 21:00:33 2006 From: andreas.brillisauer at hetzner.de (Andreas Brillisauer -- Hetzner Online AG) Date: Wed, 01 Feb 2006 11:00:33 +0100 Subject: [netflow-tools] Does softflowd open a new flow for same IPs but different ports? In-Reply-To: <43D81152.30609@mindrot.org> References: <1138204574.3442.44.camel@neptune> <43D81152.30609@mindrot.org> Message-ID: <1138788033.3465.11.camel@neptune> Damien Miller wrote: > Not at present, but it could be added pretty easily. > > Please try the attached patch. It may not apply cleanly against a > released version of softflowd - if this is the case, please try a > snapshot from http://www2.mindrot.org/softflowd_snap/ I tried to patch the following snapshots softflowd-SNAP-20060201.tar.gz softflowd-SNAP-20060131.tar.gz softflowd-SNAP-20060130.tar.gz But I get the same error with all three snapshots regarding softflowd.c ---8<------------------------------------------------------------------ debian:~/softflowd/softflowd-SNAP-20060130# patch softflowd.c softflowd.c.patch patching file softflowd.c Hunk #1 FAILED at 538. Hunk #2 FAILED at 1376. Hunk #3 FAILED at 1422. Hunk #4 FAILED at 1677. 4 out of 4 hunks FAILED -- saving rejects to file softflowd.c.rej ---8<------------------------------------------------------------------ Greetings, Andreas These are your attached patches: > Index: softflowd.8 > =================================================================== > RCS file: /var/cvs/softflowd/softflowd.8,v > retrieving revision 1.16 > diff -u -p -r1.16 softflowd.8 > --- softflowd.8 10 Jan 2005 01:50:07 -0000 1.16 > +++ softflowd.8 25 Jan 2006 23:59:34 -0000 > @@ -40,6 +40,7 @@ > .Op Fl p Ar pidfile > .Op Fl c Ar ctl_sock > .Op Fl L Ar hoplimit > +.Op Fl T Ar track_level > .Op bpf_program > .Sh DESCRIPTION > .Nm > @@ -196,6 +197,24 @@ Sets the IPv4 TTL or the IPv6 hop limit > will use the default system TTL when exporting flows to a unicast host. > When exporting to a multicast group, the default TTL will be 1 > (i.e. link-local). > +.It Fl T Ar track_level > +Specifies what flow elements > +.Nm > +should be used to define a flow. > +.Ar track_level > +may be one of: > +.Dq full > +(track everything in the flow, the default), > +.Dq proto > +(track source and destination addresses and protocol), or > +.Dq ip > +(only track source and destination addresses). > +Selecting either of the latter options will produce flows with less information > +in them (e.g. TCP/UDP ports will not be recorded). > +This will cause flows to be consolidated, reducing the quantity of output > +and CPU load that > +.Nm > +will place on the system at the cost of some detail. > .El > .Pp > Any further commandline arguments will be concatenated together and > Index: softflowd.c > =================================================================== > RCS file: /var/cvs/softflowd/softflowd.c,v > retrieving revision 1.87 > diff -u -p -r1.87 softflowd.c > --- softflowd.c 25 Jan 2006 23:25:04 -0000 1.87 > +++ softflowd.c 25 Jan 2006 23:50:06 -0000 > @@ -538,6 +538,19 @@ process_packet(struct FLOWTRACK *ft, con > if (frag) > ft->frag_packets++; > > + /* Zero out bits of the flow that aren't relevant to tracking level */ > + switch (ft->track_level) { > + case TRACK_IP_ONLY: > + tmp.protocol = 0; > + /* FALLTHROUGH */ > + case TRACK_IP_PROTO: > + tmp.port[0] = tmp.port[1] = 0; > + tmp.tcp_flags[0] = tmp.tcp_flags[1] = 0; > + /* FALLTHROUGH */ > + case TRACK_FULL: > + break; > + } > + > /* If a matching flow does not exist, create and insert one */ > if ((flow = FLOW_FIND(FLOWS, &ft->flows, &tmp)) == NULL) { > /* Allocate and fill in the flow */ > @@ -1363,6 +1376,8 @@ init_flowtrack(struct FLOWTRACK *ft) > FLOW_INIT(&ft->flows); > EXPIRY_INIT(&ft->expiries); > > + ft->track_level = TRACK_FULL; > + > ft->tcp_timeout = DEFAULT_TCP_TIMEOUT; > ft->tcp_rst_timeout = DEFAULT_TCP_RST_TIMEOUT; > ft->tcp_fin_timeout = DEFAULT_TCP_FIN_TIMEOUT; > @@ -1407,20 +1422,21 @@ usage(void) > { > fprintf(stderr, "Usage: %s [options] [bpf_program]\n", PROGNAME); > fprintf(stderr, "This is %s version %s. Valid commandline options:\n", PROGNAME, PROGVER); > - fprintf(stderr, " -i interface Specify interface to listen on\n"); > - fprintf(stderr, " -r pcap_file Specify packet capture file to read\n"); > - fprintf(stderr, " -t timeout=time Specify named timeout\n"); > - fprintf(stderr, " -m max_flows Specify maximum number of flows to track (default %d)\n", DEFAULT_MAX_FLOWS); > - fprintf(stderr, " -n host:port Send Cisco NetFlow(tm)-compatible packets to host:port\n"); > - fprintf(stderr, " -p pidfile Record pid in specified file (default: %s)\n", DEFAULT_PIDFILE); > - fprintf(stderr, " -c pidfile Location of control socket (default: %s)\n", DEFAULT_CTLSOCK); > - fprintf(stderr, " -v 1|5|9 NetFlow export packet version\n"); > - fprintf(stderr, " -L hoplimit Set TTL/hoplimit for export datagrams\n"); > - fprintf(stderr, " -6 Track IPv6 flows, regardless of whether selected \n" > - " NetFlow export protocol supports it\n"); > - fprintf(stderr, " -d Don't daemonise\n"); > - fprintf(stderr, " -D Debug mode: don't daemonise + verbosity + track v6 flows\n"); > - fprintf(stderr, " -h Display this help\n"); > + fprintf(stderr, " -i interface Specify interface to listen on\n"); > + fprintf(stderr, " -r pcap_file Specify packet capture file to read\n"); > + fprintf(stderr, " -t timeout=time Specify named timeout\n"); > + fprintf(stderr, " -m max_flows Specify maximum number of flows to track (default %d)\n", DEFAULT_MAX_FLOWS); > + fprintf(stderr, " -n host:port Send Cisco NetFlow(tm)-compatible packets to host:port\n"); > + fprintf(stderr, " -p pidfile Record pid in specified file (default: %s)\n", DEFAULT_PIDFILE); > + fprintf(stderr, " -c pidfile Location of control socket (default: %s)\n", DEFAULT_CTLSOCK); > + fprintf(stderr, " -v 1|5|9 NetFlow export packet version\n"); > + fprintf(stderr, " -L hoplimit Set TTL/hoplimit for export datagrams\n"); > + fprintf(stderr, " -T full|proto|ip Set flow tracking level (default: full)\n"); > + fprintf(stderr, " -6 Track IPv6 flows, regardless of whether selected \n" > + " NetFlow export protocol supports it\n"); > + fprintf(stderr, " -d Don't daemonise\n"); > + fprintf(stderr, " -D Debug mode: don't daemonise + verbosity + track v6 flows\n"); > + fprintf(stderr, " -h Display this help\n"); > fprintf(stderr, "\n"); > fprintf(stderr, "Valid timeout names and default values:\n"); > fprintf(stderr, " tcp (default %6d)", DEFAULT_TCP_TIMEOUT); > @@ -1661,6 +1677,19 @@ main(int argc, char **argv) > case 't': > /* Will exit on failure */ > set_timeout(&flowtrack, optarg); > + break; > + case 'T': > + if (strcasecmp(optarg, "full") == 0) > + flowtrack.track_level = TRACK_FULL; > + else if (strcasecmp(optarg, "proto") == 0) > + flowtrack.track_level = TRACK_IP_PROTO; > + else if (strcasecmp(optarg, "ip") == 0) > + flowtrack.track_level = TRACK_IP_ONLY; > + else { > + fprintf(stderr, "Unknown flow tracking level\n"); > + usage(); > + exit(1); > + } > break; > case 'L': > hoplimit = atoi(optarg); > Index: softflowd.h > =================================================================== > RCS file: /var/cvs/softflowd/softflowd.h,v > retrieving revision 1.8 > diff -u -p -r1.8 softflowd.h > --- softflowd.h 5 May 2005 03:31:42 -0000 1.8 > +++ softflowd.h 25 Jan 2006 23:39:53 -0000 > @@ -66,6 +66,11 @@ struct STATISTIC { > double min, mean, max; > }; > > +/* Flow tracking levels */ > +#define TRACK_FULL 1 /* src/dst/addr/port/proto 5-tuple */ > +#define TRACK_IP_PROTO 2 /* src/dst/proto 3-tuple */ > +#define TRACK_IP_ONLY 3 /* src/dst tuple */ > + > /* > * This structure is the root of the flow tracking system. > * It holds the root of the tree of active flows and the head of the > @@ -81,7 +86,8 @@ struct FLOWTRACK { > > /* Stuff related to flow export */ > struct timeval system_boot_time; /* SysUptime */ > - > + int track_level; /* See TRACK_* above */ > + > /* Flow timeouts */ > int tcp_timeout; /* Open TCP connections */ > int tcp_rst_timeout; /* TCP flows after RST */ From djm at mindrot.org Sat Feb 11 22:28:31 2006 From: djm at mindrot.org (Damien Miller) Date: Sat, 11 Feb 2006 22:28:31 +1100 (EST) Subject: [netflow-tools] Does softflowd open a new flow for same IPs but different ports? In-Reply-To: <1138788033.3465.11.camel@neptune> References: <1138204574.3442.44.camel@neptune> <43D81152.30609@mindrot.org> <1138788033.3465.11.camel@neptune> Message-ID: On Wed, 1 Feb 2006, Andreas Brillisauer -- Hetzner Online AG wrote: > Damien Miller wrote: > > Not at present, but it could be added pretty easily. > > > > Please try the attached patch. It may not apply cleanly against a > > released version of softflowd - if this is the case, please try a > > snapshot from http://www2.mindrot.org/softflowd_snap/ > > I tried to patch the following snapshots > > softflowd-SNAP-20060201.tar.gz > softflowd-SNAP-20060131.tar.gz > softflowd-SNAP-20060130.tar.gz > > But I get the same error with all three snapshots regarding softflowd.c Try tonight's snapshot, I just committed the change. -d From andreas.brillisauer at hetzner.de Tue Feb 14 03:15:16 2006 From: andreas.brillisauer at hetzner.de (Andreas Brillisauer -- Hetzner Online AG) Date: Mon, 13 Feb 2006 17:15:16 +0100 Subject: [netflow-tools] Does softflowd open a new flow for same IPs but different ports? In-Reply-To: References: <1138204574.3442.44.camel@neptune> <43D81152.30609@mindrot.org> <1138788033.3465.11.camel@neptune> Message-ID: <1139847316.6098.25.camel@neptune> Damien Miller wrote: > Try tonight's snapshot, I just committed the change. I just tried snapshot softflowd-SNAP-20060214.tar.gz but the -T option doesn't work. Here is what I did: ---8<------------------------------------------------------------------ # /usr/local/sbin/softflowd -i eth2 -t maxlife=300 -m 8388608 -T ip -n 127.0.0.1:9000 /usr/local/sbin/softflowd: invalid option -- T Invalid commandline option. [...] ---8<------------------------------------------------------------------ Greetings, Andreas -- Hetzner Online AG Industriestr. 6 D-91710 Gunzenhausen Tel: +49 9831 610061 Fax: +49 9831 610062 E-Mail: info at hetzner.de http://www.hetzner.de From djm at mindrot.org Tue Feb 14 07:47:34 2006 From: djm at mindrot.org (Damien Miller) Date: Tue, 14 Feb 2006 07:47:34 +1100 (EST) Subject: [netflow-tools] Does softflowd open a new flow for same IPs but different ports? In-Reply-To: <1139847316.6098.25.camel@neptune> References: <1138204574.3442.44.camel@neptune> <43D81152.30609@mindrot.org> <1138788033.3465.11.camel@neptune> <1139847316.6098.25.camel@neptune> Message-ID: On Mon, 13 Feb 2006, Andreas Brillisauer -- Hetzner Online AG wrote: > Damien Miller wrote: > > Try tonight's snapshot, I just committed the change. > > I just tried snapshot softflowd-SNAP-20060214.tar.gz but the -T option > doesn't work. oops, please apply this patch: Index: softflowd.c =================================================================== RCS file: /var/cvs/softflowd/softflowd.c,v retrieving revision 1.88 diff -u -p -r1.88 softflowd.c --- softflowd.c 11 Feb 2006 11:27:38 -0000 1.88 +++ softflowd.c 13 Feb 2006 20:46:42 -0000 @@ -1641,7 +1641,7 @@ main(int argc, char **argv) ctlsock_path = DEFAULT_CTLSOCK; dontfork_flag = 0; always_v6 = 0; - while ((ch = getopt(argc, argv, "6hdDL:i:r:f:t:n:m:p:c:v:")) != -1) { + while ((ch = getopt(argc, argv, "6hdDL:T:i:r:f:t:n:m:p:c:v:")) != -1) { switch (ch) { case '6': always_v6 = 1; From djm at mindrot.org Sun Feb 26 15:53:02 2006 From: djm at mindrot.org (Damien Miller) Date: Sun, 26 Feb 2006 15:53:02 +1100 (EST) Subject: [netflow-tools] Announce: flowd-0.9 Message-ID: <20060226045302.677CA17E620@mail.mindrot.org> Hi, This is to announce a new release of the flowd NetFlow collector, flowd-0.9. This release includes some major functionality and performance improvements. Please note that this release also changes the on-disk flow log format (it is possible to convert old flowd logs to the new format using the flowd-reader tool). The new version is available from http://www.mindrot.org/flowd.html SHA1 (flowd-0.9.tar.gz) = 54cb8ecaaa36d2f25105156170680842123b965d Thanks to the many people who suggested improvements, reported bugs and tested patches or snapshots. Some of the higlights of this release are: Storage format -------------- The on-disk storage format has been improved and given a new major number (version 3). The new format is faster to read from disk and will be more graceful to extend in the future. In particular, it is possible to add fields to this new format whilst retaining backwards compatibility. To convert logs from previous versions of flowd, please use flowd-reader's "-L" option. For example, "flowd-reader -Lqo new.log old.log" will convert the flows in "old.log" to the new format and store them in "new.log". The new format also supports some additional fields and extends the widths of some existing ones: - Receive time is now recoded as seconds and microseconds - Add Netflow V.9 source_id field - Interface indices (if_ndx_in/out), engine_type, engine_id and the source and destination AS numbers are now 32-bits wide Performance enhancements ------------------------ Flowd has had several improvements to improve its performance on busy networks or when confronted with sudden bursts of flows. The improvements are: - Addition of a basic input queue to flowd, so it is now able to cluster network packets reads - Addition of an output buffer, so every flow received doesn't end up as a tiny write to the filesystem - Increased the UDP socket receive buffer size, so the kernel can buffer more flow packets during bursts - Shrink the UDP socket send buffer size on the listening socket, because flowd never sends packets from it Python API ---------- The flowd Python API has been rewritten in 100% C, improving its performance by an order of magnitude. This rewrite changes the API and adds several new interfaces, including an iterator interface to flow logs. Reading every flow in a flow log is now as simple as: import flowd flow_log = flowd.FlowLog("flows.log", "rb") for flow in flow_log: print flow.format() Other additions to the Python API include the addition of a Flow.has_field() function as well as interval_time() and iso_time() time conversion functions flowd-reader ------------ Added the ability to read and convert flow logs from pre-0.9 versions of flowd ("flowd-reader -L"). Changed flowd-reader's output ("-o") behaviour to *overwrite* existing log files instead of *appending* to them. Please take care if you were depending on the previous behaviour. Added a "head" mode to flowd-reader. E.g. "flowd-reader -H 1024 flowd.log" will display only the first 1024 flow records. Live flow reporting ------------------- Addes support for relaying serialised flows to a local Unix domain datagram socket in realtime. This allows real-time monitoring of flows by a tiny application. A sample Python client that receives and prints the flows is provided as tools/sockclient.py. Please see the "logsock" option in flowd.conf(5) for more details. Note that this option is still considered experimental. Flow Filtering -------------- It is now possible to filter by date/time. For example: accept tag 1 after date 200601010000 before date 200601072359 Other fixes and improvements ---------------------------- Added RPM spec and init files for SuSE Linux contributed by alshu AT tut.by Allow reading from standard input in samply Python statistics program (tools/stats.py) Fix byte swapping of src/dst AS and interface indices, spotted and fix tested by Gijs Molenaar Fix bug that broke filtering on address family in flows, spotted by Gijs Molenaar Support devices that send multiple templates in a single template packet section. Thanks again to Gijs Molenaar for packet dumps that demonstrated this